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Although  the  improvement  of  equipment  systems  for  human  operation 
and  maintenance  has  been  stressed  for  many  years,  the  actual  measurement 
and  specification  of  human  performance  reliability  in  concrete  terms  has 
been  largely  ignored,  A set  of  computer  simulation  models,  which  assess 
the  human  performance  reliability  of  a system  while  the  system  is  in  the 
early  design  stage,  was  previously  developed.  The  results  of  trial  applica- 
tion of  these  simulation  models  to  an  actual  system  which  is  under  develop- 
ment are  presented.  Additionally,  a set  of  guidelines  is  presented  which  can 
form  the  basis  for  a human  performance  reliability  demonstration  in  future 
Navy  systems. 


TABLE  OF  CONTENTS 


Trends  Influencing  Hole  of  Human  1 

Automation  2 

Technological  Change 2 

Human  Reliability  2 

Purpose  of  the  Present  Work  4 

Results  of  Present  W'ork  6 

REFERENCES ') 

APPENDIX  A - Report  on  Application  of  1 to  3 Man  Model 11 

APPENDIX  B - Report  on  Application  of  4 to  20  Man  Model 4 f) 

APPENDIX  C - Human  Reliability  Demonstration  Guidelines 115 


APPLICATIONS  OF  HUMAN  PFR FOR  MANCF  READABILITY 
EVALUATION  CONCEPTS  AND  DEMONSTRATION  GUIDELINES 


> 


I 


) 


i 


* 


> 


Advancing  technology  and  changes  in  the  role  of  the  human  in  advanced 
equipment  systems  have  served  to  reemphasize  the  importance  of  minimizing 
human  error  in  system  function.  For  example,  in  many  advanced  military 
systems,  decision  making  in  short,  time  critical  situations  places  a heavy 
burden  on  the  system  operator.  The  operator  often  no  longer  possesses  the 
luxury  of  time  when  he  needs  it  most,  and  he  often  has  time  to  waste  when 
he  needs  time  least.  In  this  regard,  Pettitt  (1974)  pointed  out  that: 

HWP-31 , the  antiship  missile  defense  doctrine, 
specifies  that  commanding  officers  should  dele- 
gate firing  authority  to  evaluators  during  a high 
threat  situation. . .COs  do  need  to  know  how  to  del 
egate  authority  to  defend  their  ships  in  rapidly 
developing  high  threat  situations. .The  necessity 
to  compress  drastically  the  time  required  for  rec- 
ognition of  a threat  through  its  evaluation,  con- 
sideration of  weapons  capabilities,  weapons  assign- 
ment and  analysis  of  weapon  performance,  until  final 
kill,  makes  it  evident  that  bold  steps  must  be  taken. 

Henceforth,  reactions  will  be  measured  in  seconds 
rather  than  minutes.  Decisions,  as  well  as  evalua- 
tions, will  undoubtedly  be  required  from  the  officer 
"on  scene"  at  the  time  the  threat  evolved,  since  time 
will  no  longer  permit  the  old  "detect-evaluate-dissem- 
inate"  routines  established  in  World  War  II. 


Trends  Influencing  Role  of  Human 


At  least  two  interrelated  trends  seem  to  affect  the  role  of  the  human 
in  advanced  systems.  One  is  the  trend  towards  automation.  The  second 
trend,  which  is  not  mutually  exclusive  from  the  first,  is  technological  ad- 
vance. Although  the  two  concepts,  automation  and  technological  change, 
possess  something  in  common,  there  is  also  a difference  between  the  two. 
Specifically,  technological  advance  implies  the  effects  of  increased  scien- 
tific and  production  capability  regardless  of  whether  or  not  the  function 
served  by  the  capability  is  automatically  or  manually  performed.  Automa- 
tion, per  se,  may  or  may  not  involve  an  increase  in  capability.  The  human 
performance  reliability  effects  of  technological  change  may  be  different 
from  those  which  result  from  automation. 
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Automation 


Automation  represents  an  attractive  means  for  reducing  personnel 
costs.  Gaites  (1974)  pointed  out  that  personnel  costs  now  represent  42  per 
cent  of  the  operating  budget  and  26  percent  of  the  total  budget  of  the  Navy. 

He  also  pointed  out  that  each  man  in  a modern  destroyer  size  ship  requires 
five  tons  of  ship  representing  500  cubic  feet.  Construction  costs  for  this 
amount  of  structure  approximate  $25,  000,  and  the  structure  must  then  be 
maintained  for  the  30  year  shelf  life  of  the  ship. 

Siegel,  Wolf,  and  Williams  (1975)  reviewed  trends  in  automation 
in  the  Navy.  Additional  treatment  of  the  data  of  Siegel,  Wolf,  and  Williams 
has  yielded  the  curves  presented  in  Figure  1,  which  presents  projected 
levels  of  automation  for  destroyer  subsystems  over  the  years.  Quite  ob- 
viously, such  projections  are  subject  to  a number  of  assumptions,  but  the 
curves  serve  to  point  up  the  potential  for  changes  in  the  role  of  the  human 
in  advanced  systems. 

Technological  Change 

One  need  not  belabor  the  technological  advances  which  have  taken 
place  over  the  past  several  decades.  These  technological  advances  have 
been  reflected  in  prior  ship  systems.  It  can  be  assumed  that  they  will  be 
further  reflected  in  new  ships  with  consequent  effects  on  human  performance. 

For  example,  consider  electronic  signal  processing  and  specifically 
signal  correlation  and  anticorrelation.  Until  recently,  signal  correlation 
and  anticorrelation  have  been  mostly  laboratory  techniques.  Simply  stated, 
a signal  correlator  looks  for  some  signal  which  possesses  a correlation 
with  a known  event.  This  correlation  is  often  temporal.  A signal  anticor- 
relator looks  for  signals  which  do  not  have  a correlation  with  a known  event. 
The  use  of  these  two  techniques  would  seem  to  have  considerable  potential 
for  applications  in  many  new  systems  such  as  IFF,  detecting  and  recogniz- 
ing signals  from  countermeasures  (both  active  and  passive),  and  validating 
the  authenticity  of  received  command  messages.  Such  systems  could,  how- 
ever, require  highly  trained  personnel  to  serve  not  as  links  in  the  signal 
processing,  but  to  perform  advanced  functions  which  if  not  completed  cor- 
rectly could  negate  system  effectiveness. 

Human  Reliability 

In  answer  to  the  pervasive  need  for  methods  for  assessing,  while 
a system  is  in  the  design  state,  whether  or  not  the  human  in  advanced  man/ 
machine  systems  will  be  able  to  perform  his  required  functions,  the  Naval 
Sea  Systems  Command  instituted  a "human  reliability"  methods  program. 
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Within  the  total  human  reliability  program,  the  Applied  Psychological  Serv- 
ices has  developed  a number  of  techniques  for  assessing  human  reliability. 
Table  1 summarizes  the  methods  and  the  sources  of  additional  information 
relative  to  these  Applied  Psychological  Services'  developments. 

Briefly,  three  related  techniques  have  been  developed  and  individu- 
ally validated.  The  first  of  these  is  a hand  calculational  method  for  predict- 
ing the  probability  that  a given  technician  will  be  able  to  perform  his  required 
functions.  The  technique  is  relatively  easy  to  apply  and  can  be  employed  with 
out  recourse  to  high  speed  calculational  equipment.  The  second  and  third  of 
these  techniques  are  based  on  computer  simulation.  Two  previously  develop- 
ed digital  simulation  models  were  expanded  and  elaborated  so  as  to  allow  hu- 
man reliability  prediction.  The  first  model  is  most  appropriate  for  evalua- 
tions of  systems  which  involve  1 to  3 technicians.  The  second  model  allows 
consideration  of  systems  which  involve  from  4 to  20  technicians.  In  addition 
to  a human  reliability  metric,  this  4 to  20  man  model  yields  measures  of:  (1) 
human  availability  and  human  mean  time  to  repair,  (2)  equipment  reliability, 
availability,  and  mean  time  to  repair,  and  (3)  system  (combined  human  and 
equipment)  reliability,  availability,  and  mean  time  to  repair. 

The  power  of  the  set  of  approaches  lies  in  the  analytic  spectrum  it 
provides.  The  hand  calculational  technique  permits  rapid,  desk  top  human 
reliability  analysis.  The  1 to  3 man  simulation  model  may  be  employed 
when  a fine  sieve  analysis  of  the  tasks  to  be  performed  is  sought.  Individu- 
al tasks  are  broken  down  into  individual  component  actions  (e.  g.  , throw  a 
switch)  to  help  bring  out  human  oriented  design  defects  and  to  quantify  them. 
This  fine  grain  simulation  model  is  most  appropriate  for  relatively  short 
duration  tasks,  and  generally  only  critical  (time  dependent)  aspects  of  mis- 
sions should  be  simulated  using  this  simulation  model.  The  4 to  20  man  mod- 
el permits  a coarser  but  possibly  more  useful  level  of  simulation.  Entire 
days  or  numbers  of  days  can  be  simulated.  This  allows  psychosocial  factors, 
stress,  level  of  aspiration,  cross  training,  and  similar  factors  to  be  consid- 
ered in  the  simulation.  Using  all  approaches  on  the  same  system  allows  a 
hierarchical  approach  with  successive  checks  and  independent  evaluations. 

Purpose  of  the  Present  Work 

While  previous  Applied  Psychological  Services'  work  developed  and 
evaluated  the  various  human  reliability  predictive  techniques,  there  has  been 
no  test  of  the  methods  and  of  their  ability  to  make  a positive  contribution 
relative  to  a system  which  is  in  the  design  phase.  The  present  work  applied 
the  two  separate  computer  simulation  models  to  the  AN/SQS-26,  LAMPS, 
AN/SQR-19  system.  At  the  time  of  the  analysis,  the  system  was  in  the  mid- 
dle of  the  detailed  design  process. 


4 


ry  of  Human  Reliability  Predictive  Methods  Developed  by  Applied  Psychological  Services 


s 


p 

P 

5 

§ 

C 

e 

1 

P 

p 

s 

a 

p 

St 

P 

u 

h 

a 

03 

0 

p 

0 

Cl 

0 

Q 

0 

> 

ft 

Q 

H 

Q 

Si 

a 

•H 

0 

03 

fit 

O 

X 

0) 

OJ 

P 

\ 

> 

0) 

> 

0) 

u 

fl 

u 

03 

Cl 

03 

Cl 

P 

•H 

X J 

*0 

c 

c 

c 

c 

C 

0 

0 

0) 

0 

03  u 

0 

0) 

0 

•H 

•H 

to 

rH 

3 ft 

ft 

3 

•H 

p 

p 

0) 

0 ^ 

p 

0* 

p 

fl 

fl 

V4 

H 

H 0 

fl 

•H 

fl 

u 

0 

ft 

fl 

fi  ft 

XJ 

C U 

T3 

•H 

•H 

B 

ft 

B ft 

•H 

i-H 

H 

u 

P 

U XJ 

H 

0 0 

Ol 

ft 

H 

•^4 

oj  c 

03 

<u  5 

fl 

E 

ft 

w 

c 

H fl 

> 

H ft 

> 

< 

< 

H 

M 

; 

(4 

4 

<4 

t4 

) 

% 

i ^ 

U3 

• 

4 

X 

* 

» 

L3 

■ 

• 

• 

• 

% 

• 

• 

% 

# 

• 

X 

• 

a 

4 M 

M 

ft 

1 M 

» 

M 

* 

< 

H 

• 

• 

H 

ft 

• 

ft 

H 

• 

ft 

« 

• 

• 

* 

X 

• 

• 

• 

X 

• 

X 

• 

• 

• 

X 

• 

• 

£ 

• 

r ** 

< 

* < 

< 

»3 

% 

< 

• 

u 

t 

< 

u 

< 

• 

< 

»-5 

< 

c 

* 

c 

» 

• 

(0 

3E 

• 

3E 

• 

£ 

• 

0 

3 * 

* 

fl 

* 

•3 

c 

* 

*-> 

i 

* 

»3 

* 

c 

» 

E 

\ *4 

H 

E 

I r- 1 

fl 

a 

ft 

ft 

ft 

c 

ft 

fl 

ft 

C 

4 V 

03 

E 

1 03 

% 

B 

0) 

% 

•H 

0) 

to 

N 

0) 

03 

>1 

03 

03 

% 

g 

03 

0)  O' 

a 

i PT 

P 

P 

P' 

ft 

rH 

P' 

A 

£ 

O' 

X) 

O'  JZ 

tn 

P' 

ft 

p 

O' 

3 0) 

0) 

XJ 

> OJ 

H 

3 

03 

rH 

ft 

03 

fl 

1 

fl 

HI  e 

0) 

fl 

03 

03 

ft 

3 

03 

a 

) -H 

•H 

$ 

l -H 

0 

fl 

•H 

0 

•H 

•H 

0) 

fl 

03 

•H 

fl 

•H 

03 

•H 

ft 

0 

fl 

ft 

ft  tn 

w 

ft 

3 

W 

2 

2 

to 

a 

w 

>4 

w 

2 

W 

2 

Ui 

XJ 

<D  * 
P >, 
fl 


f: 


c fl 

ft  ft 

g'S 

•H 

ft  W 

o 6 
03 


■g 


ft  >1 
fl  ft 

Wi  -H 


s a 


p 

u 


'O 

o> 

ft 

u 

cj 


o o 

«ft  ft 

>1  x 
p cx 

•H  w 


vfi  « 
CN  v0 
I r*» 
W o 
OH 
(/) 


ft 

w 

Cl 

o a> 

p ft 
i 


•o 

> 

TJ 

p 

X3 

p 

fl 

ft 

z 

P o< 

u r* 

T3 

0;  C 

* 

fl  ft 

03 

w 

0) 

w 

XJ 

A 

< 

ft  w 

C r- 

C 

Cl  0 

P 

P 

Cl 

>. 

Cl 

>1  ft 

fl 

ft  \ 

fl  O' 

fl 

ft  ft 

ft 

ft  0 

X 

w 

X 

</> 

ft 

ft 

X3 

ft  z 

E ft 

P 

ft 

O ft 

fl 

ft 

c 

A < 

c 

P 

fl 

ft 

X3 

u 

03 

Cl 

03 

> 

fl 

fl 

0 * 

c 

P U 

A 

C 03 

0 

c 

0 

c 

X 

ft  *0 

ft  in 

0) 

ft  ft 

fl 

O Ci 

ft 

•H 

ft 

ft 

ft 

10 

ft  c 

Cl  03 

£ 

ft  ft 

C 

•H  X 

.c 

X 

03 

Cl 

CM 

CJ  fl 

03  C 

ft 

ft  ft 

ft 

p 

ft 

u 

ft 

u 

0 

1 

X 

X ft 

0 

A ft 

fl 

fl  * 

0) 

fl 

03 

0 

p 

m 

% 

ft 

ft 

fl  < 

P 

XJ  P 

XJ 

£ 

fl 

£ 

£ 

fl 

0 * 

C U) 

C 03 

0> 

•h 

c 

ft  ft 

Q 

1 

fl1 

Q 

1 

Cl 

W \D 

fl  X 

fl  T3 

> 

ft  Cl 

ft 

ft  f— 1 

£ 

c 

r-> 

c 

• 

03  \ 

E 2 

E ft 

03 

03  0 

fl 

fl  ft 

fl 

O' 

fl 

H 

X 

Z O' 

3 a 

3 3 

a 

X ft 

£ 

> .a 

< 

£ 

ft 

< 

£ 

M 

0 

3 ft 

X 3 

X 0 

tr 

c 


in  * 

ft  >i 
03  P 
•0  ft 

M 

0 *H 
ft 

* QJ 

h x 

1 § 

* I 

< X 


>1 

p 


O'  ft 

in 

c 

z w 

p 

3 

ft 

03  ft 

* 

03 

a e 

fl 

tn 

ft 

p A 

in 

fl 

u 

Cl 

fl 

•H 

* 

c fl 

r- 

w 

03  P 

p 

Q3  * 

A 

♦ 

ft  ft 

O' 

Cl 

x in 

tn 

£ « 

fl 

« 

o 
ft 

o o' 

X ft 

1 1 

X P 

oj 

fl  £ 

O'  c 
C 0 
ft  ft 
Q<  P 

o o 

ft  *H 
0)  TJ 
> 03 
03  Cl 
Q X 


T3  H 
C 03 
fl  XJ 

e>  § 

•h  e 

•o  0 
c ft 

3 P 


8JII 


8 .a 

to 

ft  c 

3 i 

ft 

fl" 

.fl  0 
O P 
U 

x *h 


c 

o 

•H 

p 

•0 

H 

3 

.an 

tn  XJ 

* i 

03 

P c 
3 fl 

to 

U <N 


0 

p 


p 

<0 

ft 

1-3 

to  XJ 

3 g 

P C 

I* 

0 o 

u CN 


0 

p 


0 

p 


3 

B ft 
ft  0) 
« TJ 

Cl  | 

0) 

P c 
3 fl 

iE 

0 o 

U CM 


XJ  O' 
c c 

fl  ft 
*0 
c c 

0 3 

3 a 

* 6 


Sfl 

i O 
o u 
u a 


w 

3 

o 

ft 

P 

fl 

> 


■P 

Ss 

O 

&: 


o 

CX 

ft: 


4^ 

Sh 

O M 
CL  to 
0)  Q ■ 

^ Cl 

£,§• 

J t/j 

S.  , to 
$•£* 
£££ 
l!  Il  II 


While  the  previously  described  techniques  allow  early  analysis  and 
assessment  of  a system  from  the  human  and  the  integrated  human/ equip- 
ment points  of  view,  a need  was  also  apparent  for  an  early  demonstration 
that  a given  level  of  human  performance  reliability  has,  in  fact,  been 
achieved  by  the  integrated  design.  An  equipment  reliability  test  is  already 
required  prior  to  the  acceptance  of  a new  Navy  system  or  subsystem.  Hu- 
man reliability,  however,  is  not  included  in  this  test.  Accordingly,  a set 
of  guidelines  for  the  conduct  of  a human  performance  reliability  demonstra- 
tion was  developed.  The  human  performance  reliability  demonstration  is 
conjectured  as  an  additional  quality  assurance  technique  which  is  imple- 
mented after  initial  fabrication  but  before  system  delivery.  As  such,  the 
demonstration  would  help  to  assure  that  the  required  human  and  integrated 
system  reliability  has  been,  in  fact,  obtained. 

Figure  2 shows  the  conceptual  relationship  between  equipment  reli- 
ability and  human  reliability.  This  scheme  starts  with  the  definition  of  the 
objectives  of  the  system  under  consideration.  After  establishing  the  objec- 
tives, the  design  of  the  system  and  the  training  required  to  operate  the  sys- 
tem are  established.  The  system  design  affects  the  training  requirements, 
equipment  reliability,  and  the  human  reliability.  Training  directly  affects 
human  reliability.  Equipment  reliability  and  human  reliability  both  interact 
to  affect  one  another  and  compound  to  produce  system  reliability.  One  im- 
portant aspect  of  this  scheme  is  the  realization  that  all  the  major  factors 
must  be  considered  together  in  order  to  yield  a total  system  which  possesses 
adequate  achieved  reliability. 

Results  of  Present  Work 


As  indicated  above,  the  present  work  possessed  three  separate  objec- 
tives. The  work  performed  and  the  end  products  of  each  effort  are  presented 
as  Appendices  A (1  to  3 man  model  application),  B (4  to  20  man  model  appli- 
cation), and  C (demonstration  guidelines)  to  the  present  report.  The  results 
of  the  two  simulations  are  similar  but  complementary,  with  the  1 to  3 man 
model  providing  specific  information  on  failures  and  their  cause,  while  the 
4 to  20  man  model  provided  indications  of  the  effect  of  these  failures  on  sys- 
tem performance. 

t 

i 

There  were  few,  if  any,  problems  associated  with  the  applications  of 
the  two  models.  About  one  day  was  involved  developing  each  scenario  simu- 
lated. The  team  which  developed  the  scenarios  was  composed  of  both  engi- 
neers who  were  familiar  with  the  system  and  human  factors  personnel  who 
were  familiar  with  the  models.  For  the  1 to  3 man  model,  about  three  days 
were  required  for  input  data  development.  Input  data  development  for  the 


4 to  20  man  model  required  about  10  working  days.  Both  model  applications 
produced  useful  data  relative  to  system  improvement.  Equipment  system 
developmental  personnel  indicated  that  the  results  made  important  contribu- 
tions to  their  thinking  and  that  the  emergent  recommendations  were  intuitive 
ly  correct.  Implementation  of  the  recommendations  emerging  from  these 
human  reliability  analyses  is  now  under  consideration.  Accordingly,  at 
least  relative  to  these  initial  tests,  the  utility,  practicality,  and  ability  of 
the  methods  to  contribute  economically  to  total  system  quality  assurance 
seems  to  have  been  demonstrated.  There  has  been  no  similar  trial  imple- 
mentation of  the  demonstration  guidelines.  Accordingly,  their  utility  and 
practicality  as  a system  design  support  tool  must  await  further  development. 
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ABSTRACT 


A computer  simulation  model  was  applied  to  the  acts  and  behaviors 
of  the  operators  of  the  combined  AN/SQS-26  and  AN/SQR-XX  sonar  system 
as  they  perform  the  sequence  of  subtasks  involved  in  completing  a 
somewhat  complex  search,  detect,  locate,  and  track  scenario  including 
target  loss  and  reacquisition.  The  results  indicated  76  per  cent 
overall  success  for  "average"  operators  and  nominal  time  allowance. 
This  success  percentage  varied  considerably  as  a function  of  simulated 
operator  proficiency  and  time  allowance.  Time  allowance  seemed  to 
exert  a greater  influence  on  success  than  simulated  personnel  pro- 
ficiency. Areas  of  highest  operator  unreliability  were  indicated  to 
be:  classification,  target  reacquisition  after  target  loss, target 

motion  analysis,  and  cognitively  oriented  tasks  such  as  speed  change/ 
course  derivation. 
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CHAPTER  I 
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INTRODUCTION 

Over  the  past  several  years,  the  Navy  Sea  Systems  Command  has 
been  concerned  with  a program  oriented  toward  the  development  of 
t measures  for  assessing  the  "human  reliability"  in  a system.  To  this 

end,  the  total  program  has  developed  both  empirical  and  simulation 

models.  Both  types  of  model  are  held  to  be  useful  for  predicting 
human  reliability  while  a system  is  in  the  early  design  stage. 

Human  reliability  is  defined  as  that  measure  of  human  perform- 
. ance  which  when  combined  with  equipment  reliability  allows  one  to 

* predict  total  system  reliability.  This  measure,  if  obtained  early 

in  the  system  development  cycle,  allows  the  reevaluation  of  equipment 
design,  operational  procedures,  and  maintenance  procedures  to  promote 
maximal  total  system  performance. 

The  present  work  applied  one  of  the  techniques  developed  within 
1 the  total  program  to  the  combined  AN/SQS-26  and  AN/SQR-XX  systems.  The 

specific  technique  employed  was  computer  simulation  of  operator 
actions  through  a previously  developed  simulation  model.  The  model 
simulates  the  acts  and  behavior  of  the  system  operators/maintainers 
as  they  complete  the  sequence  of  subtasks  necessary  for  task  comple- 
tion. The  model's  output  provides  information  relative  to  total  task 
success,  time  for  task  completion,  subtasks  failed  within  the  total 
task,  periods  of  stress  on  the  operator/maintainer , and  the  like. 

The  model  is  called  the  Siegel-Wolf  simulation  model  and  has  been 
described  in  detail  previously.  However,  for  purposes  of  continuity 
the  model  is  summarized  below. 


Description  of  Siegel-Wolf  Model 

The  Siegel-Wolf  model  is  used  with  a high-speed,  general 
purpose  digital  computer.  To  employ  the  model,  the  system  designer 
or  evaluator  makes  an  analysis  of  the  man-machine  system  and  the 
task  under  consideration.  The  performance  of  each  operator  is 
arranged  into  ordered,  discrete  actions  called  "subtasks,"  and  for 
j.  each  of  these  certain  source  data  are  compiled.  These  data, 

together  with  selected  parameter  values  (e.g.,  the  time  allotted 
for  task  performance)  are  introduced  into  the  digital  computer. 

The  computer  sequentially  simulates,  according  to  the  rules  of  the 
model,  the  "performance"  of  each  subtask  by  each  operator.  The 
normal  sequence  of  subtasks,  whether  linear  or  non-linear,  may  be 
modified  if  actions  have  to  be  skipped  or  repeated  due  to  failure 
t by  either  operator  at  a subtask  or  as  a result  of  operator  decisions. 

A simulation  is  completed  when  the  operators  either  use  all  allotted 
time  or  successfully  complete  the  task.  During  the  course  of  the 
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computer's  performance"  of  the  task,  results  are  recorded  indicating 
the  areas  of  operator  overload,  failure,  idle  time,  peak  stress, 
etc.,  for  the  given  set  of  selected  parameters.  Repetitions  of  the 
simulation,  with  different  parameter  values,  yield  a range  of 
records.  If  the  results  indicate  modifications  to  the  design  of  the 
system,  new  designs  may  be  similarly  tested  to  determine  the  extent 
of  improvement  brought  about  by  the  modifications. 

The  other  data  required  by  the  computer  in  advance  of  the 
simulation  are  the  parameters  and  initial  conditions.  These  permit 
the  adjustments  of  critical  variables  and  the  consequent  determina- 
tion of  the  range  of  their  effects.  In  order  to  simulate  intra-  and 
inter-individual  differences  of  performance,  the  simulation  of  any 
individual  subtask  is  based,  in  part,  on  a random  process.  Because 
of  this  stochastic  effect,  it  is  necessary  to  repeat  the  simulation 
of  a task  many  times  in  order  to  obtain  sufficient  performance  data 
for  each  set  of  conditions.  The  parameter  N is  used  to  indicate  the 
number  of  times  a given  task  is  to  be  simulated;  there  are  N simulations 
(or  N iterations)  per  computer  'run'. 

To  use  the  model  twelve  items  of  input  data  are  required  for 
each  subtask  ( i * 1,2, ... ,n)  and  each  operator  (j-  1,2).  These  data 
may  be  derived  from  such  procedures  as  task  analysis,  literature 
search,  or  personal  interviews.  The  required  input  data  for  each 
operator  are: 

1.  average  subtask  execution  time,  £-  • : the  average 

time  required  by  the  jth  operator  to  perform  sub- 
task i.  This  average  value  represents  the  case 
in  which  the  operator  is  under  no  stress. 

2.  average  standard  deviation,  o.  .;  taken  around  the 
t £ . for  the  average  operator  wnile  not  under 
stress. 

3.  average  sub  task  probability  of  success,  p*  •:  the 
probability  that  the  average  operator,  J,  while  not 
under  stress,  can  perform  subtask  i successfully. 

4.  indication  of  subtask  essentially,  E^:  an  in- 

dicator specifying  whether  or  not  the*' successful 
performance  of  subtasr  i by  operator  j is  essential 
to  successful  completion  of  the  task.  This  datum 
allows  the  computer  to  identify  and  ignpre  non- 
essential  subtasks  during  "highly  urgent"  conditions. 
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idle  time  requirement,  l^j:  the  point  in  time 

before  which  operator  j Is  not  permitted  to  begin 
subtask  i . 

indication  of  whether  subtask  i is  a decision 
aubtask  or  a normal  action  subtask. 

subtask  number,  (i,j)f  to  be  performed  next  by  an 
operator  j if  he  fails  at  subtask  i or  if  he 
chooses  the  first  of  two  alternative  courses  in 
a decision  subtask. 

subtask  number , (i, j)8  to  be  performed  by  an 
operator  j if  he  succeeds  on  subtask  i , or  chooses 
the  second  alternative  course  in  a decision  sub- 
task . 

subtask  number,  d^j  (mnemonic  delay)  which  must  be 
successfully  completed  by  his  partner  before  an 
operator  j can  begin  subtask  i.  By  proper  selection 
of  dij  values,  it  is  possible  to. cause  either 
operator  to  "wait"  until  his  partner  has  completed 
a stipulated  subtask  successfully.  Thus,  "waiting" 
for  one's  partner  is  simulated  differently  from  time 
spent  "idling"  until  a fixed  time  as  in  5 above. 

indication  of  whether  or  not  subtask  i for  operator j 
is  a special  subtask  in  which  the  operators  communicate 
with  each  other. 

time,  ffj,  required  to  perform  all  remaining  essential 
subtasks  (including  i)  at  average  execution  times, 
assuming  no  failure.  With  no  branching  or  decisions: 


n _ 

l t. 


i=  1 


ij’ 


time,  T^q,  required  to  perform  all  remaining  non- 
essenticu  subtasks  (including  i)  at  average  execution 
times,  assuming  no  failures. 


Three  pairs  of  parameters  may  be  varied  from  run  to  run  in 
order  to  evaluate  either  the  model  on  a man-machine  system.  The 
stress  thresholds,  Mj,  one  for  each  operator,  may  be  considered  as  the 
operator's  "breaking  point."  For  example,  an  Mj  value  of  2.0  in- 
dicates that  the  operator  begins  to  become  slower  and  less  accurate 
at  the  point  at  which  he  has  more  than  twice  as  much  to  do  (at 
average  speed)  as  he  has  time  available.  Prior  to  this  point,  any 
added  backlog  of  essential  subtasks  induces  stress  which  affects 
the  operator's  actions  so  that  they  become  faster  and  more  accurate. 

The  parameters,  Tj,  are  the  total  times  allotted  to  each  operator  for 
performance  of  the  whole  task.  For  a two  man  team,  the  task  is  considered 
to  have  been  successfully  completed  if  both  operators  complete  all 
required  subtasks  within  the  time  specified  by  the  larger  of  the  two 
values. 

The  parameters,  Fj,  /hich  account  for  variance  among  individuals, 
are  termed  the  individuality  factors  for  the  two  operators.  F-  is  a 
multiplicative  factor  with  a value  of  unity  for  the  average  operator . 
For  faster  or  more  highly  motivated  operators  (Fj  < 1),  and  for  slower 
operators  (Fj  > 1). 
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The  Simulation  Sequence 

Having  stored  the  program,  parameters,  and  initial  conditions, 
the  computer  begins  processing  the  data  sequentially.  It  determines 
the  sequence  of  subtasks  to  perform  in  accordance  with  (i,o)8  and 

input  data.  Its  determination  of  which  operator  to  simulate 
at  any  given  time  in  the  sequence  depends  upon  T.L,  the  total  time 
used  by  operator  j while  "performing"  all  subtasks'7  from  the  start  of 
the  simulation  through  subtask  i-1.  The  operator  having  the  smaller 
Tii  value  is  selected,  and  his  next  subtask  is  simulated. 

I'd 

If  waiting  is  required,  the  sequence  continues  using  data  for 
the  other  operator.  Then,  a determination  is  made  as  to  whether  the 
operator  must  idle  until  an  amount  of  timely  has  elapsed  from  the 
beginning  of  the  simulation.  If  idling  is  required,  the  idle  time 
- T^j  is  recorded,  totals  accumulated,  set  equal  to  I.. , and 
ttfe  control  returned  to  determine  which  operator  to  simulate ‘next . 

If  no  idling  is  required,  a determination  is  made  of  whether  or  not 
subtask  i is  a communication  subtask.  If  it  is,  the  operators  are 
synchronized  by  setting  the  total  time  used  by  both  at  that  of  the 
one  who  has  taken  longer.  This  may  result  in  a wait  for  either 
operator  and  is  treated  as  the  wait  described  above. 


Urgency  and  Stress 

Following  the  synchronization  of  the  operators,  or  if  the 
subtask  is  not  a communication  subtask,  one  of  three  states  of 
"urgency"  is  determined  for  each  operator,  based  upon  the  remaining 
time  available  to  him  for  completing  the  task  and  the  average  time 
required  to  complete  it  if  no  failures  occur: 


1.  The  situation  is  non-urgent  when  sufficient  time 
remains  to  complete  all  remaining  subtasks. 

2.  The  urgent  state  occurs  if  the  time  available  is 
insufficient  to  complete  all  remaining  essential 
subtasks. 

3.  The  situation  is  highly  urgent  if  there  is  in- 
sufficient time  available  for  completing  even 
the  remaining  essential  subtasks. 


In  the  urgent  and  highly  urgent  conditions,  the  computer  ignores  the 
non-essential  subtasks. 


Following  the  determination  of  the  degree  of  "urgency!1  the 
b tree a condition  is  calculated.  Current  psychological  theory  suggests 
that  emotion  or  stress  acts  as  an  organizing  agent  on  behavior 
up  to  a certain  point,  and  beyond  it  as  a disorganizing  agent.  Ac- 
cordingly, the  model  recognizes  an  organizing  effect  on  operator 
performance  as  long  as  Bij  (the  stress  upon  operator  j just  prior 
to  his  performance  of  subtask  i)  is  less  than  a threshold  value  Mj: 
if  eij  exceeds  Mj,  the  effect  is  disorganizing.  During  non-urgent 
and  urgent  conditions  ajj  is  defined  as  equal  to  unity;  when  the 
situation  is  highly  urgent,  stress  is  defined  as  the  ratio  of  the 
sum  of  the  average  execution  times  for  the  remaining  essential  sub- 
tasks  to  the  total  time  remaining: 


eijm  m„jT 

Tj  - nj 


In  other  words,  stress  is  the  ratio  of  how  much  is  left  to  do  to 
the  amout  of  time  available  in  which  to  do  it. 

Since  each  operator  has  an  individual  time  limit  on  his  per- 
formance and  a task  failure  occurs  only  when  the  larger  of  these 
limits  is  exceeded,  it  is  possible  for  the  simulation  to  continue 
with  one  operator  (arbitrarily  selected  as  operator  1)  having 
exceeded  his  limit.  Should  this  be  the  case,  the  stress  condition 
of  this  operator  is  set  equal  to  his  threshold  value,  Mj,  for  the 
remainder  of  the  simulation. 


Subtask  Success  and  Failure 


The  model  assumes  that  the  actual  probability  of  successful 
performance  of  a given  subtask,  pijt  is  a function  of  pijt  8^,  and 
Mj,  as  follows: 


Pij 


* ^ (1  * Pij}  (eij  - I* 

Pij  + ———————— 

M.  - 1 

3 

Pij  (eij  + 1 ~ Mj)  + (Mj  - Bij) 


if  Bij  < Mj 

if  Mj  <_  Bij  <_  Mj  + 1 


2Pij  - 1 


if  Bij  > Mj  + 1 
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Thus  the  probability  of  success  increases  linearly  with  stress 
from  a value  of  until  it  assumes  a value  of  unity  at  the  stress 
threshold.  Following  this  point,  the  probability  assumes  the 
average  value,  after  which  it  decreases  linearly  until,  when 
stress  has  a value  equal  to  Mj  + 1,  it  levels  off  at  a value  which 
is  decreased  lrom  p. • by  an  amount  equal  to  1 - Pij.  In  order  to 
determine  actual  success  or  failure  for  any  subtask,  the  computer 
generates  a pseudo-random  number,  R2 , uniformly  distributed  over 
the  unit  interval  from  flg.  The  operator  is  considered  to  have  per- 
formed the  subtask  successfully  if  R,  is  less  than  otherwise 

he  is  assumed  to  have  failed.  This  Implies  that  there  will  be  a 
failure  with  probability,  in  the  long  run. 


Other  Details  of  the  Simulation  Process 


An  operator  may  find  it  desirable,  or  external  conditions  may 
require  him,  to  select  one  of  several  alternative  courses  of  action. 

The  decision  subtask , incorporated  to  enable  such  branching,  skipping, 

* and  looping,  causes  the  computer  to  select  the  next  subtask  with- 

out "consuming  operator  time."  Decision  subtasks  may  be  placed  any- 
where in  the  sequence.  For  these,  and  essentiality  have 

no  meaning.  The  calculation  is  bypassed  and  the  last  pseudo- 
random number,  tf,,  from  the  previous  subtask  is  compared  against  the 
Plj  of  the  decision  subtask.  Therefore,  the  next  subtask  to  be  per- 
| formed  as  a result  of  the  decision,  is  subtask  ( i,j)s  with  probability 

or  subtask  with  probability  1 - p 

In  certain  subtasks,  such  as  placing  a cursor  on  a target, 
several  trials  for  the  same  action  are  usually  required  although  a 
single  action  may  occasionally  be  successful.  These  subtasks  are  treated 
by  the  computer  as  requiring  a single  control  action  with  a relatively 
I low  probability  of  success.  The  probability  of  success  on  a single 

trial  is  determined,  using  the  formula  that  if  p is  the  probability 
of  a success  on  a single  trial  and  p*  is  the  probability  of  at  least 
one  success  after  n trials  then  p=  1 - n ✓ ( 1 - p* ) . 

Operators  are  assumed  to  remember  and  execute  the  correct 
. sequence  of  subtasks.  However,  the  possibility  of  one  or  both 

* operators  neglecting  a subtask  or  rearranging  the  performance  may 

be  studied  by  additional  runs  using  the  different  sequences  concerned. 

A change  in  the  predetermined  sequence  of  subtasks  in  the  event  of 
emergency  can  be  dealt  with  by  establishing  special  "danger" 
sequences  to  be  simulated. 

t 


t 
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Validation  of  the  Siegel-Wolf  Model 

The  Siegel-Wolf  model  has  received  a wide  variety  of  valida- 
tions and  verifications  and  possesses  a history  of  use  relative  to 
sonar  systems.  In  the  empirical  validations  the  model's  output 
has  been  compared  with  real  life  data.  Examples  of  these  valida- 
tions are  reported  in  Federman  and  Siegel  (1973),  Siegel  and 
Macpherson  (1967),  Siegel  and  Wolf  (1963),  and  Siegel,  Wolf,  and 
Sorenson  (1962).  In  all  of  these  validations,  reasonable  conformity 
between  the  model's  predictions  and  real  life  criterion  data  has 
been  found. 
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The  Present  Simulations 


Systems  Simulated 


The  system  simulated  in  the  present  model  application  is  a new 
sonar  system  combining  the  best  features  of  two  systems — the  AN/SQS-26 
and  the  AN/SQR-XX  sonar  systems.  The  new  system  allows  an  increase 
in  capability  through  sensor,  signal  processing,  and  information 
display/processing  improvement.  The  new  system  will  provide  capability 
in  a wide  variety  of  modes  with  many  operator  options  for  a wide 
variety  of  situations  and  circumstances.  A detailed  description  of 
the  capabilities  of  the  new  system  is  not  given  here  because  of 
"security"  considerations. 


The  Mission  Simulated 

As  a test  of  the  ability  of  the  operators  of  the  combined 
AN/SQS-26  and  AN/SQR — XX  sonar  system  to  complete  the  required 
subtasks  in  a typical,  time  bound  scenario,  a 24  minute  "mission" 
was  selected.  The  scenario  requires  a reasonable  variety  of  operator 
actions  and  is  performed  under  reasonably  high  stress  conditions. 

The  scenario  was  derived  in  consultation  with  personnel  at  the  Navy 
Underwater  Systems  Center,  New  London,  Conn.  Figure  1-1  pictorially 
presents  a time  line  history  of  the  tracks  of  the  vessels  during  the 
derived  scenario.  At  the  start  (T  + 0),  own  ship  is  on  a 045° 
course  at  a speed  of  20  knots.  At  this  time,  two  consoles  are  manned 
(one  operator  at  each). 

In  order  to  fix  the  target  ship,  the  AN/SQS-26  operator  requests 
a course  change.  Our  ship  changes  course  to  a 090°  heading  at  T + 5 
minutes.  This  course  change  consumes  about  two  minutes.  After  the 
course  change,  the  target's  location  is  determined  and  these  data 
are  entered  into  the  computer.  At  T + 10,  the  target  ship  stops  dead 
in  the  water  and  the  AN/SQS-26  operator  recommends  a new  heading 
(000°). 

At  T + 14  minutes,  the  target  ship  begins  a hard  turn  to  port 
and  own  ship  reacts  by  turning  to  045°  at  T + 15  minutes.  At  T + 18 
minutes,  own  ship  reduces  speed  to  10  knots.  Target  contact  is  lost 
at  T + 20  minutes  at  which  time  the  target  ship  passes  directly 
under  own  ship.  The  final  simulated  subtasks  are  target  reacquisi- 
tion . 


The  sequence  of  subtasks  performed  by  each  operator  during  this 
time  interval  is  presented  in  Tables  1-1  and  1-2.  The  subtask 
sequence  is  placed  on  a time  line  in  Figure  1-2. 

Figure  1-3  presents  the  task  data  of  Tables  1-1  and  1-2  as 
employed  in  the  simulation. 
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Table  1-1 


Sequence  of  AN/SQS-26  Operator  Subtasks  Simulated  (Operator  1) 


1. 

Acknowledge  detection  report 

18. 

Project  track 

2. 

Select  sit.  sum. 

19. 

Slew  cursor 

3. 

Report  sonar  contact 

20. 

Recommend  projected  course 

4. 

Locate  target  on  PNB 

21. 

Ship  turning-2  minute  wait 

5. 

Report  bearing 

22. 

Note  change  in  bearing  rate 

6. 

Determine  course  change 

23. 

Respond  to  XX  report 

7. 

Recommend  course  change 

24. 

Recommend  intercept  course 

8. 

Select  fine  bearing 

25. 

Report  new  intercept  course 

9. 

Direct  fine  bearing 

26. 

Derive  speed  change 

10. 

Ship  turn  - 2 minute  wait 

27. 

Recommend  speed  change 

11. 

Slew  curser  to  measure  bearing 

28. 

Acknowledge  report  lost  contact 

12. 

Read  digital  display 

29. 

Report  to  CONN 

13. 

Reply  to  XX  bearing  report 

30. 

Change  bearing  CRT  to  section  PNB 

14. 

Input  26  bearing  to  computer 

31. 

Search  for  target 

15. 

Input  XX  bearing  to  computer 

32. 

Acknowledge  report  recontact 

16. 

Read  digital  range  estimate 

33. 

Report  to  CONN 

17. 

Report  to  CONN 

Table  1-2 


Sequence  of  AN/SQR-XX  Operator  Subtasks  Simulated  (Operator  2) 


1. 

Report  detection 

13. 

Slew  cursor  to  measure  bearing 

2. 

Slew  cursor 

14. 

Read  digital  display 

3. 

Tag  target 

15. 

Report  bearing  to  supervisor 

4. 

Select  single  beam  on  lower  CRT 

16. 

Classify  target 

5. 

Assign  DEMON 

17. 

Notice  intervening  target 

6. 

Slew  cursor 

18. 

Report  interfering  target 

7. 

Select  bearing  correction 

19. 

Report  classification 

8. 

Select  harmonic  cursor 

20. 

Decide  contact  loss 

9. 

Measure  harmonics 

21. 

Report  lost  contact 

10. 

Acknowledge  fine  bearing  instruction 

22. 

Select  vector 

11. 

Select  fine  bearing 

23. 

Search 

12. 

Call  up  CRT  profile 

24. 

Report  contact 
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Figure  1-3. 


Task  analytic  input  data. 
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CHAPTER  II 


SIMULATIONS  COMPLETED,  RESULTS,  AND  DISCUSSION 


On  the  basis  of  the  scenario  and  operator  actions  described  in 
Chapter  I,  a number  of  simulations  were  completed  in  order  to  determine 
operator  reliability  in  the  combined  AN/SQS-26  and  AN/SQR-XX  sonar 
system.  Fifteen  simulation  runs  of  100  iterations  each  were  completed. 
Within  the  15  runs,  operator  teams  at  five  proficiency  levels  (speeds) 
were  simulated  at  three  total  time  allowances.  The  1440  second 
(24  minute)  time  allowance  is  considered  to  be  the  nominal  case  with 
a shorter  (1380  seconds)  and  a longer  (1500  seconds)  timelimit  re- 
presenting "harder"  and  "easier"  situations  respectively.  Operator 
proficiency  combinations  ranging  from  reasonably  "above  average"  to 
well  "below  average"  were  included.  The  proficiency  value  equal  to 
1.0  represents  the  "average"  team  with  values  above  1.0  representing 
poorer  proficiency  and  values  below  1.0  representing  better  proficiency. 
Table  2-1  presents  the  parameters  varied  in  the  various  simulation 
runs. 


Table  2-1 


Run 

No. 

1 

2 

3 

4 

5 

6 

7 

8 
9 

10 

11 

12 

13 

14 

15 


Simulation  Runs  Completed 


Operator  1 Operator  2 

Proficiency  (F)  Proficiency  (F) 


Time  Allowed 
(Secs. ) 

1440. 

1440. 

1440. 

1440. 

1440. 

1380. 

1380. 

1380. 

1380. 

1380. 

1500. 

1500. 

1500. 

1500. 

1500. 
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Per  Cent  Success 


Figure  2-1  presents  the  percentage  of  success  for  the  simulated 
operators  as  a function  of  team  proficiency  when  the  time  allowed 
varied  from  23  minutes  to  25  minutes.  The  overall  effect  of  reducing 
the  time  allowed  for  completion  of  all  essential  subtasks  was  clearly 
to  reduce  the  percentage  of  successful  task  completions.  This  effect 
seems  quite  reasonable.  Figure  2-2,  which  presents  the  mean  per- 
centage of  success  at  each  time  allowance,  indicates  a close  linear 
relationship  between  time  allowed  and  mean  percentage  success.  In- 
creasing the  time  allowance  from  23  to  25  minutes  caused  the  success 
percentage  to  increase  from  55  per  cent  to  94  per  cent. 

The  obtained  mean  success  percentage  as  a function  of  simulated 
team  proficiency  level  is  shown  in  Figure  2-3.  The  simulated  results 
suggested  overall  success  percentage,  in  terms  of  the  criteria 
employed,  of  about  76  per  cent  for  an  "average"  (F=  1.0)  operator 
team.  Increasing  the  team  proficiency  from  F=  1.2  (low  proficiency) 
to  F=  0.9  (high  proficiency)  caused  about  an  eight  per  cent  increase 
in  success.  Accordingly,  for  the  time  allowance  range  simulated  and 
the  team  proficiency  range  considered,  time  allowance  exerted  a 
greater  effect  on  success  percentage  than  team  proficiency. 


Total  Work  Time 


Total  time  used  (actual  work  time)  represents  another  criterion 
which  may  be  used  as  an  index  of  operator  performance.  This  measure 
sometimes  reflects  changes  in  system  performance  when  percentage 
success  appears  to  be  insensitive. 

Figure  2-4  presents  the  simulation  results  relative  to  the 
simulated  subtask  sequence  as  a function  of  time  allowed.  With  a 
smaller  amount  of  time  allowed,  we  would  expect  the  team  to  come 
under  stress,  work  faster,  and  complete  all  essential  subtasks 
earlier.  This  anticipated  trend  was  indicated.  The  23  minute  time 
allowance  (1380  seconds)  provided  a noticeable  drop  in  time  used, 
while  there  was  only  slight  difference  in  total  time  used  between  the 
24  and  the  25  minute  time  allowances.  The  latter  indication  seems 
to  suggest  that  the  24  minute  time  allowance  was  sufficient  most  of 
the  time.  Accordingly,  the  24  minute  allowance  induced  onlyslightly 
more  stress  than  the  25  minute  condition.  An  F=  0.90  team  is 
theoretically  about  30  per  cent  more  proficient  than  a F=  1.2 
operator  team.  The  simulation  results  indicated  only  about  a 1 per 
cent  difference  in  total  time  used  for  these  two  types  of  teams. 


(HIGH)  .9  1.0  1.1  1.2  (LOW) 

TEAM  PROFICIENCY 

FIGURE  2-1.  PERCENTAGE  OF  SUCCESS  WITH  THREE  LEVELS  OF  TIME  ALLOWED 
AND  FOUR  TEAM  PROFICIENCY  LEVELS 
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FIGURE  2-3.  PERCENTAGE  OF  SUCCESS  AT  FOUR  OPERATOR  SPEEDS 
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The  small  size  of  this  difference  seems  to  rest  on  the  fact  that  the 
mission  scenario  included  a number  of  time  waits  for  own  ship 
maneuver  response.  There  was  enough  waiting  time  so  that  the  effects 
of  operator  proficiency  were  masked  when  total  time  used  formed  the 
basis  for  the  comparison. 


Task  Failure  Frequency 

The  subtasks  failed  most  frequently  by  the  simulated  operator 
team  provide  insight  into  areas  of  potential  system  improvement. 
Equipment  changes  or  improved  training  can  often  improve  system 
performance  once  the  problem  areas  are  identified.  The  three  most 
frequently  failed  subtasks  for  operator  1 are  presented  in  Table 


2-2. 

Subtasks  Fai led 

Table  2-2 

Most  Frequently  by  Operator  1 

Subtask 

Number 

Times  Failed/100 
Iterations 

DescriDtion 

31 

240 

Search  for  target  (after  loss  of  contact ) 

4 

111 

Locate  target  on  PNB 

26 

47 

Derive  speed  change 

The  two  tasks  which  were  most  frequently  failed  by  operator  1 in- 
volved signal  detection. 

The  third  most  frequently  failed  subtask  was  "derive  speed 
change."  The  subtask  involves  slightly  complex  cognitive  activity, 
as  we  understand  the  subtask.  The  process  of  performing  this 
calculation  might  be  reviewed  from  the  point-of-view  of  job  aids 
which  might  increase  its  success  probability  on  the  first  attempt. 

The  three  subtasks  failed  most  frequently  by  operator  2 are 
presented  in  Table  2-3. 
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Table  2-3 


I Subtasks  Failed  Most  Frequently  by  Operator  2 

Subtask  Times  Failed/100 


Humber 

Iterations 

Description 

13 

47 

Slew  cursor  to  measure  bearing 

2 

47 

Slew  cursor 

16 

43 

Classify  target 

For  operator  2,  the  tracking  subtasks,  along  with  the  target 
classification  subtask,  were  the  high  failure  frequency  of  subtasks. 
The  classification  subtask  is  known  to  be  quite  difficult  and  seems 
an  area  where  computer  guiding  might  provide  significant  failure 
savings. 

While  subtask  failure  frequency  is  important  in  allowing  in- 
sight relative  to  where  errors  may  be  made  by  an  operator,  the  time 
lost  in  repeating  or  touching  up  tasks  failed  provide  insight  into 
consequences  of  these  errors.  The  subtasks  with  the  greatest 
amount  of  repetition  time  due  to  failure  for  operator  1 are  presented 
in  Table  2-4. 


Table  2-4 


High  Fai lure 

(Repetition)  Time  Subtasks  for  Operator  1 

Task 

Failure 

■> 

Number 

Time* 

Description 

31 

2324.21 

Search  for  target  (after  loss  of  contact) 

22 

1589.12 

Note  change  in  bearing  rate 

4 

809.93 

Locate  target  on  PNB 

26 

410.69 

Derive  speed  change 

24 

257.39 

Recommend  intercept  course 

*Times 

are  across  100 

iterations . 

The  subtasks  which  resulted  in  the  most  repetition  time  due  to 
failure  on  the  first  attempt  by  operator  1 were  those  which  re- 
quire signal  detection  and  target  motion  analysis.  The  latter  is 
an  obvious  area  in  which  job  aids  might  be  helpful. 
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The  subtask  failures  involving  the  highest  repetition  times 
for  operator  2 are  shown  in  Table  2-5. 


Table  2-5 

High  Failure  (Repetition)  Time  Subtasks  for  Operator  2 


Task 

Number 

Failure 

Time''* 

Description 

16 

4183.20 

Classify  target 

23 

181.83 

Search 

2 

178.03 

Slew  cursor 

13 

174.62 

Slew  cursor  to  measure  bearing 

6 

154. 55 

Slew  cursor 

9 

129.16 

Measure  harmonies 

*Timea  are 

total  aoroBB 

100  iterations. 

The  target  classification  subtask  was  suggested  by  the  simula- 
tion to  be  the  most  salient  in  terms  of  contributing  to  time  con- 
sumption because  of  subtask  failure.  This  finding  is  concordant  with 
the  finding  relative  to  failure  frequency. 
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Peak  Stress 


An  examination  of  the  subtasks  on  which  the  highest  stress 
occurs  can  also  provide  insight  into  the  critical  points  of  a 
scenario.  The  simulation  induced  peak  stress  points  for  operator  1 
are  presented  in  Table  2-6. 


Table  2-6 

Peak  Stress  Subtaskq  for  Operator  1 


Team  Pro-  Time  Allowed  (Seconds) 


f iciency 

1380 

1440 

1500 

.9 

32,31* 

32,31 

_ 

1.0 

33,32,  31,22 

32,32,22 

- 

1.1 

32,31 

32,31,26,22 

32 

1.2 

32,31 

32,33,27,22 

32,31 

*Subtasks  on  which  peak  stress  occurred  in  5 per  cent  of  the  iteration  or  more. 


Peak  stress  for  operator  1 generally  was  indicated  to  occur 
towards  ttie  conclusions  of  the  scenario  simulated,  i.e.  , after 
target  loss  and  during  the  attempt  at  target  reacquisition.  These 
data  are  in  accordance  with  the  prior  suggestion  relative  to  job 
aiding  for  this  stage  of  the  scenario.  Subtasks  22  (note  change 
in  bearing  rate),  26  (derive  speed  change)  and  27  (recommend  speed 
change)  also  indicated  peak  stress  under  the  two  shorter  time 
allowances.  These  data  are  also  in  conformity  with  prior  indica- 
tions. In  the  25  minute  time  allowance,  the  F=  0.9  and  F=  1.1 
operator  teams  evidenced  no  stress  at  all.  Accordingly  no  peak 
stress  is  reported  for  these  conditions. 

Operator  2 indicated  a similar  peak  stress  pattern.  The  peak 
stress  tasks  for  operator  2 are  presented  in  Table  2-7. 


Table  2-7 


Peak 

Stress  Subtasks  for  Operator  2 

Team  Pro 

Time  Allowed  (Seconds) 

ficiency 

1380 

1440 

1500 

.9 

24,23* 

24,16 

16 

1.0 

24,23 

24,19,18,16 

24,17 

1.1 

24.23 

24,19,16 

24,16 

1.2 

24,23 

24,19,18,16 

24,18,16 

*Subtasks  on  which  peak  stress  occurred  in  S per  cent  of  the  iterations  or  more. 


Subtasks  23  and  24,  which  seem  to  predominate  in  the  Table 
2-7  analysis  are  concerned  with  target  reacquisition  after  target 
loss.  This  was  the  same  scenario  segment  for  which  peak  stress  was 
identified  for  operator  1.  Subtask  19  (report  classification)  was 
next  most  salient  in  this  analysis  and  subtask  16  (classify  target) 
followed. 

Subtask  18  (report  interfering  target)  and  subtask  16  (notice 
target)  also  tended  to  be  stress  inductive  during  certain  simulations. 
In  summary,  for  operator  2 and  for  the  scenario  simulated,  the  target 
recognition  tasks  indicated  the  most  stress  along  with  subtasks  and 
considerations  concerned  with  the  interfering  target.  These  stress 
points  seem  reasonable  and  in  accordance  with  logical  expectancy. 


Waiting  Time 

Within  the  scenario  simulated,  considerable  operator  time  was 
spent  in  waiting  for  reports  from  a partner  and  for  the  ship  to 
complete  a maneuver.  The  percentage  of  mission  time  spent  waiting 
for  the  F*  1.0  ("average")  operator  team  is  presented  in  Table  2-8. 

Table  2-8 

Waiting  Time  Percentage  for  Various  Time  Allowances 

Time  Allowed  (Seconds) 

1380  1440  1510 

58%  56%  &3% 

78%  73%  72% 


Operator 

1 

2 
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Roughly  55  per  cent  of  the  time  of  operator  1 was  spent  wait- 
ing and  45  per  cent  involved  actual  work.  For  operator  2,  waiting 
time  involved  75  per  cent  of  the  total.  During  this  "waiting  time" 
the  crewman  may  be  performing  nonscheduled  or  nonessential  subtasks 
such  as  monitoring  displays.  However,  in  the  present  simulation, 
it  seems  that  the  crew  workload  was  low.  This  indication  opens  the 
possibility  of  one  operator  maning,  at  least  from  the  point  of  view 
of  the  scenario  simulated. 


Discussion 

The  combined  AN/SQS-26  and  AN/SQR-XX  sonar  system  was  suggested 
by  the  computer  simulation  to  be  an  effective  and  functional  concept, 
at  least  for  the  scenario  simulated. 

The  primary  simulation  results  indicated  a 76  per  cent  pre- 
dicted average  mission  success  when  24  minutes  were  allowed  for 
completing  all  required  operator  actions.  When  the  time  allowed 
was  decreased  to  23  minutes,  only  55  per  cent  success  was  indicated. 
Increasing  the  time  allowed  to  25  minutes  yielded  a 94  per  cent 
predicted  success  rate. 

Subtask  failure  frequency,  subtask  repetition  time,  and  peak 
stress  subtask  analyses  were  in  general  conformity  and  suggested  the 
most  system  debilitating  subtasks  to  be  those  concerned  with  target 
acquisition  after  target  loss',  classification,  target  location  on 
PNB,  and  cognitively  oriented  subtasks  such  as  speed  change  deriva- 
tion and  intercept  course  derivation.  Design  emphasis  in  these 
areas  may  be  indicated  along  with  job  aid  provisions.  Alternatively, 
these  may  represent  areas  for  training  emphasis.  There  is  also  the 
possibility  that  a "predictive"  type  of  display,  for  use  during 
target  reacquisition,  would  exert  a pronounced  effect  on  total  task 
success.  Similarly,  "computer  guiding"  during  classification  might 
be  helpful. 

The  simulation  results  did  not  suggest  that,  for  the  scenario 
simulated,  the  operators  are  under  high  time  pressure.  These  data, 
when  viewed  in  conjunction  with  the  Figure  1-2  time  line  analysis 
suggest  that  single  operator  manning  (given  appropriate  equipment 
design)  may  be  possible. 

Certainly,  only  one  scenario  was  simulated  in  the  present 
analysis.  Simulation  of  operator  performance  for  other  typical 
and  atypical  conditions  may  be  indicated. 
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Summary  and  Conclusions 


A computer  simulation  model  was  employed  to  simulate  the  acts 
of  the  operators  of  the  combined  AN/SQS-26  and  AN/SQR-XX  system  as 
they  perform  the  sequence  cf  subtasks  involved  in  a complex  scenario 
involving  search,  detection,  track,  loss  of  contact,  and  target 
reacquisition  with  interference  from  a secondary  target.  Simulations 
of  the  performance  of  "average,"  "above  average,"  and  "below  average" 
teams  was  completed  when  various  time  quotas  are  allowed  for  comple- 
tion of  all  "essential"  subtasks.  For  the  scenario  simulated,  the 
following  conclusions  seem  indicated: 

1.  For  "average"  operators  and  in  terms  of  the  success 
criterion  employed,  a 76  per  cent  success  rate  was 
indicated  or  about  a 75  Der  cent  system  degradation 
due  to  operator  unreliability. 

2.  Time  allowance  seemed  to  exert  a greater  effect  on 
success  rate  than  simulated  variation  in  operator 
proficiency. 

3.  The  subtasks  which  seemed  to  contribute  most  to 
system  unreliability  as  a function  of  operator  un- 
reliability were  those  subtasks  concerned  with  target 
reacquisition  after  contact  loss,  classification, 
target  location  on  PNB,  and  cognitively  oriented 
subtasks  such  as  speed  change/course  derivation. 

Design  and/or  training  emphasis  may  be  indicated  in 
these  areas. 

4.  At  least  for  the  scenario  and  subtask  sequence  in- 
cluded, a relatively  low  workload  was  indicated. 

The  possibility  of  one  operator  manning  remains 
open. 
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ABSTRACT 


A multiman  human  reliability  simulation  model  was  used  to 
predict  the  operator  performance  in  the  combined  AN/SQS  -26,  LAMPS, 
and  AN/SQR-19  system.  The  simulated  scenario  involved  prosecution  of 
five  targets,  two  of  which  were  classified  as  threats,  over  a four  hour 
period.  For  the  selected  scenario,  a variety  of  simulations  was  completed. 
The  various  runs  involved  modifications  of  crew  qualifications,  work  pace, 
and  motivational  conditions. 

The  results  indicated  that, regardless  of  crew  qualifications  or  moti- 
vational level,  reasonably  high  performance  values  can  not  be  anticipated. 
Additional  design  emphasis  on  the  operator/equipment  interfaces  and  on  the 
between  human  component  information  transfer  was  suggested  as  the  method 
increasing  predicted  system  performance  level. 
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CHAPTER  I 


INTRODUCTION  AND  MODI  U • SCRIPTION 


In  1975,  the  need  for  measurement  of  human  reliability  provoked 
the  realization  that  a computer  simulation  model  developed  earlier  at 
Applied  Psychological  Services  possesses  many  of  the  necessary  qualifi- 
cations for  preproduction  estimation  of  human  reliability  in  man/ machine 
systems.  The  model  considers  the  performance  parameters  which  affect 
human  reliability  including  human,  equipment,  and  environmental  consi- 
derations. Throughout  this  report  the  model  is  called  the  SW4-20  model 
after  the  model's  developers  (Siegel  & Wolf)  and  the  number  of  operators/ 
maintainers  of  a system  (between  4 and  20)  which  the  model  can  simulate. 
The  earlier  developed  SW4-20  model  was  accordingly  elaborated  to  include 
the  calculations  necessary  for  providing  the  measures  shown  in  Table  1.  A 
number  of  other  modifications  were  also  added  to  the  prior  stochastic 
simulation  model.  These  include  but  were  not  limited  to: 

1.  Operator  induced  equipment  failures 

2.  Equipment  performance  measures 

3.  Shift  simulation  capabilities 

4.  Increased  consumable  consideration 
including  spare  parts 

5.  Data  summarization  on  equipment, 
human,  and  system  reliability,  avail- 
ability, and  mean  time  to  repair 


Description  of  SW4-20  Model 

The  model  in  its  current  form  is  basically  a sequential  processor 
with  incorporated  human,  equipment,  and  environmental  factors.  To  employ 
the  model,  the  events  to  be  performed  by  the  system  operators  during  a 
simulated  mission  are  described  through  input  data.  The  model's  program 
organizes  the  events  in  sequence  according  to  their  prerequisite  event  and 
serial  components.  The  sequencing  allows  branching  for  proportional  in- 
clusion of  less  likely  tasks.  Having  organized  the  work,  the  model  assigns 
appropriate  crew  members  to  each  task  and  sequentially  simulates  the 
performance  of  the  events  by  the  simulated  operators.  After  pei  formance 
of  all  events  scheduled  has  been  simulated  or  the  allowed  time  has  elapsed, 
the  mission  is  considered  to  be  completed  and  output  statistics  reflecting 
performance  quality  are  compiled.  One  simulation  of  a mission  is  called  an 
iteration.  Due  to  the  stochastic  nature  of  many  of  the  model's  features, 
reliable  results  require  a number  of  iterations  of  a mission.  These  itera- 
tion results  are  averaged  to  yield  final  (run)  results. 
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The  model  allows  specification  of  complete  output  detail  for  every 
event  performed,  summary  statistics  on  each  iteration,  overall  run 
statistics,  or  any  combination  of  these. 

The  details  of  the  model  (including  model  flow,  variable  descrip- 
tions, computational  logic,  and  output  descriptions)  are  found  in  Siegel,  Wolf, 
and  Lautman  (1974)  and  in  Siegel,  Wolf,  and  Fischl  (1969), 

Logic  Flow  of  SW4-20  Model 

A summary  flow  chart  of  SW4-20  model  is  presented  as  Figure  1. 

The  majority  of  the  model  input  is  entered  in  NAMELIST  form.  The 
NAMELIST  form  allows  flexibility  in  the  degree  of  precision  of  the  input 
data  and  was  thought  to  be  most  appropriate  for  this  type  of  model.  The 
second  block  of  the  input  (Figure  1)  shows  the  initialization  of  the  simulated 
crew's  proficiency,  and  motor  capabilities  parameters.  The  task  and 
situational  parameters  which  are  to  be  represented  during  the  simulation 
(including  the  emergencies,  equipment  failures,  and  illnesses)  are  also 
entered  during  this  initialization. 

Entry  point  "b"  in  Figure  1 leads  to  the  factors  which  are  considered 
prior  to  the  simulation  of  each  event  or  task  processed  during  the  simulated 
mission.  The  earliest  time  or  starting  time  of  an  event  is  determined  as  a 
function  of  available  crew,  prerequisite  task  completion,  and  equipment 
availability.  Whether  or  not  to  perform  the  event  at  all  is  determined  on 
the  basis  of  consumable  availability,  time  available,  task  priority,  and 
personnel  available.  The  physical  condition  of  the  participating  crew 
members  in  terms  of  time  since  last  sleep  and  physical  fatigue  is  also 
considered  and  evaluated  before  the  actual  task  performance  is  simulated. 

Having  selected  the  tasks  and  the  sequence  of  tasks  whose  performance 
is  to  be  simulated  this  day,  the  model  proceeds  with  the  simulation  of  one 
day's  performance  of  the  crew.  This  is  completed  on  a task-by-task  basis. 
Then,  the  work  of  the  next  day  to  be  simulated  is  organized  and  simulated. 

At  the  completion  of  the  simulation  of  all  days  in  the  mission,  the  results 
are  summarized.  Additional  simulations  of  the  same  mission  are  then 
performed  in  the  same  manner  and  the  results  of  the  individual  simula- 
tions (iterations)  are  summarized  into  run  results.  A number  of  factors  in- 
fluence the  performance  of  the  simulated  crew.  These  are  summarized  in 
Table  2 which  presents  the  major  variables  considered  by  the  model  and 
the  output  provided.  The  use  of  each  variable  in  the  current  simulation  is 
shown  in  the  right  hand  column  of  Table  2. 


START 


SEQUENCE 
TASKS  BASED 
ON  PREREQUISITES 


DETERMINE 
NUMBER  AND 
TIMES  OF 
EQUIPMENT 
BREAKDOWNS 


SIMULATE  TASK 

• COMPUTE  DURATION 

• DETERMINE  SUCCESSOR 

FAILURE 

• UPDATE  CONSUMABLE 

USAGE 

• DETERMINE  PERFORMANCE 

ADEQUACY 

• INCREMENT  COUNTERS 


YES 


MORE 
EVENTS  TO 
SIMULATE? 


BEFORE  PROCESSING  TASKS 
DETERMINE 

• SHOULD  TASK  BE  SKIPPED? 

• NUMBER  OF  MEN  REQUIRED 

• EARLIEST  START  TIME 

• SPECIALITIES  REQUIRED 

• MEN  AVAILABLE 

• GROUP  LEADER 

• TIMES  AVAILABLE 

• PHYSICAL  CONDITION 

- SLEEP 

- FATIGUE 

- CALORIES 

• PSYCHOLOGICAL  CONDITION 

- STRESS 

- ASPIRATION 


COMPUTE  ITERATION  DATA 

• AVERAGE  PHYSICAL  WORKLOAD 

• PRIMARY  COMPETENCE 

• EQUIPMENT  PERFORMANCE 

• EQUIPMENT  EFFICIENCY 

• SAFETY  INDEX 

• MENTAL  LOAD  INDEX 


COMPUTE  MISSION  SUMMARY 

• HUMAN  RELIABILITY 

• HUMAN  AVAILABILITY 

• HUMAN  MTTR 

• EQUIPMENT  RELIABILITY 

• EQUIPMENT  AVAILABILITY 

• EQUIPMENT  MTTR 


• SYSTEM  RELIABILITY 

• SYSTEM  AVAILABILITY 

• SYSTEM  MTTR 


FIGURE  1.  OVERVIEW  OF  FLOW  IN  SW4-20  STOCHASTIC  SIMULATION  MODEL. 
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Major  SW4-20  Model  Variables.  Output,  and  Present  Simulation  Conditions 


t 


t 


t 


PAtUILF 


neSFHT  tutu L AT ICH  COHPITtOHS 


Quantity 


P 


Categoriea/types 


I 


1-20  awi 

groups 

«ro«p  leader 


4 M 

1 froop 
supervisor 


prinary/secondary  specialties  4 primary  ipKtilttN  -supervisor,  M.  WW,  1* 
10  personnel  specialties  ajvl 
cross  training  Ho  era*  training 

command  achalon 


R Goal* 

a 


aspiration  normal,  high,  and  lew 
loader' a expectation  normal,  high,  and  lew 
performance  adequacy 


0 Physical 
N 

N 

1 


physical  workload 
Motion  sicknaae 
haaard  (safety  indoa) 

slasp 

physical  incapability 
(sickness) 
physical  workload 


no  physical  incapacitation 

Bin  lawn  to  none 

low 

rostsd  craws 

none 

Bin  taw 


L Performance 
Attributes 


competence 

fatigue 

Pace 

stress  and  strasa  threshold 

mental  load 

unsunned  station  hours 


complete  and  averaga 

no  fatigue 

average  and  above  average 
averaga 

high  mental  load 


* 


H 

I 

8 

8 

I 

O 


Composition 

Duration 

feivironmmit 

llments 

(tasks) 


200  event  par  day  of  10  typos 

hours  to  10  days 
shifts 


sea  state 

essentiality 

typos  (scheduled,  unary  an  cy, 
rapalr) 

praeodanco  (task  and  tins) 

performance  tins 

flmad  and  variable  event  tines 


N 


touch  up  or  repeat 
completion  time  limit 


112  sequential  tasks,  21  types 
1 four  hour  shift 

ass  state  1,2 

all  tasks  essential 

Repair  and  emergency  events  included 

task  and  time  proraquialtaa 

variable  event  times  only 

toudnpe  and  repeats  allowed 
time  limits  whenever  appropriate 


Quantity 

K 

Q Capability 
0 
I 
P 

M Performance/ 
E status 
N 
T 


Mission  ef- 
fectiveness 


Tine  Utilisation 


10  types 


1 types  - 1 consoles 


failure  and  generation  of 

repairs  repelrs  to  1 consoles 

operator  initiated  failures 

failure  rates  failure  rates  included 

up  time 
down  tine 
performance  level 

consumables  levels  no  eeasunabloa  used 


system  reliability  level  evaluated 

ayst*  performance  level  most  evaluated 

equipment  performance  efficiency 
system  global  effectiveness 
level 

possum ableu  balances 
equipment  and  human  MTRf  4 MTTR 

success,  idle,  sleep,  repair,  all  evaluated 

no.  of  evonts,  success,  fail. 

Ignore,  primary,  secondary 


Personnel 


performance  adequacy 
physical  and  montal  load 
health  and  safety  indices 
performance 


evaluated 

included 

health  and  safety  not  evaluated 
considered 


Report  frequency 


event,  day,  si union 
iterations,  and  run 


full  detail  on  all  runs 
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Development  and  Validation  of  SW4-20  Simulation  Model 


The  SW4-20  simulation  model  was  first  developed  by  Applied 
Psychological  Services  under  contract  with  the  Office  of  Naval  Research 
(Siegel,  Wolf,  & Fischl,  1969).  Since  its  original  development,  the  model 
has  been  elaborated  on  and  been  the  subject  of  a number  of  validational 
investigations.  The  most  recent  work  on  the  model, including  its  extension 
into  reliability,  maintainability,  and  availability, have  been  supported  by 
the  Navy  Sea  Systems  Command. 

Reasonable  and  useful  degrees  of  correspondence  have  been  found 
between  simulation  output  and  actual  Fleet  data  in  a number  of  actual 
situations  including  Viet  Nam  river  patrols  (Siegel,  Wolf,  & Cosentino, 
1971),  trust  territory  reconnaissance  by  patrol  gunboat  class  Navy  ships 
(Siegel,  Lautman,  & Wolf,  1972)  and  sonar  systems  operating  on  DE1052 
(Knox)  class  destroyer-escort  vessels  (Siegel,  Wolf,  & Williams,  1976). 
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CHAPTER  II 


THE  PRESENT  SIMULATION 

The  present  report  describes  the  methods  and  results  of  application 
of  the  SW4-20  model  to  the  acts  and  behaviors  of  a sonar  team  operating 
the  combined  AN/SQS-26,  LAMPS,  AN/SQR -19  system  during  an  attack 
situation  which  places  a heavy  load  on  the  simulated  sonar  team. 


The  Mission  Simulated 


Within  the  present  simulation,  the  system  is  assumed  to  be  manned 
by  four  persons--a  sonar  supervisor  and  three  operators.  Each  simulated 
operator  is  assigned  to  one  console.  The  consoles  are  assumed  to  be 
organized  into  a complete  sonar  suite  with  information  transferred  from  one 
console  to  another  by  manual  means. 

Own  ship  is  assumed  to  be  on  a high  speed  transit  and,  over  the 
course  of  the  work  simulated,  five  targets  are  processed.  The  sequence 
of  events,  as  simulated,  is  shown  in  Figure  2.  As  indicated  in  Figure  2, 
a first  target  (target  A)  is  detected  15  minutes  into  the  scenario  (t  + 15) 
at  which  time  it  is  processed.  At  t + 22,  a second  target  (target  B)  is 
detected.  Both  targets  are  lost  as  they  leave  the  second  convergent  zone 
(60,  000  yards).  At  t + 58  and  t + 85,  target  C and  D are  respectively 
detected  and  processed.  At  t + 107  target  A enters  the  first  convergent  zone 
and  is  reacquired  at  t + 111.  Target  D enters  the  first  convergent  zone 
and  is  detected  and  processed  starting  at  t + 135.  At  t + 150,  a fifth  target 
is  detected  (target  E)  in  the  second  convergent  zone  and  target  B is  shortly 
thereafter  reacquired  in  the  first  convergent  zone.  At  t + 185,  target  C 
is  reacquired  in  the  first  convergent  zone,  and  the  decision  is  reached  to 
attack  target  C.  The  scenario  ends  with  the  localization  (and  attack)  of 
target  C. 


A major  portion  of  the  sonar  team's  activities  is  repeated  for  each 
target.  The  sequence  is  always  initiated  with  target  detection  by  the  SQR-19 
operator.  The  initial  detection  is  assumed  to  require  approximately  five 
minutes  with  a standard  deviation  of  two  minutes.  Following  the  initial 
contact,  the  LAMPS  console  operator  switches  to  beam  mode  and  then  enters 
the  target  bearing  and  range  data  on  the  situation  summary  report.  At 
this  point,  the  supervisor,  who  has  been  observing  the  SQR-19  and  the  LAMPS 
operator  actions,  decides  whether  or  not  to  classify  the  target,  and  whether 
the  SQS-26  operator  should  be  ordered  to  slew  and  search  the  first  con- 
vergent zone.  In  the  nominal  case,  it  is  assumed  that  the  SQS-26  operator 


FIGURE  2 (Cont. 
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will  be  ordered  to  slew  and  he  will  begin  to  do  so.  The  LAMPS  console 
operator  is  assumed,  at  this  time, to  begin  to  classify  the  target  and  the 
SQR-19  operator  is  assumed  to  assign  DEMON  and  the  fine  bearing 
cursor.  The  supervisor  will  at  this  time  log  the  target  contact  data. 

The  preceding  activities  describe  the  procedures  assumed  during 
the  nominal  target  detection  and  classification  where  the  target  has  not 
reached  the  first  convergent  zone  or  been  classified  as  a threat. 

In  the  case  of  a target  found  to  be  in  the  first  convergent  zone,  the 
spectral  characteristics  of  the  target  are  assumed  to  be  used  by  the 
supervisor  to  attempt  to  identify  which,  if  any,  of  any  previously  identified 
targets  is  involved  or  whether  a new  target  is  involved.  If  no  new  target 
is  involved,  the  supervisor  then  orders  the  LAMPS  operator  to  merge  the 
two  targets.  The  LAMPS  operator,  when  so  ordered,  is  assumed  to  set 
up  for  data  entry  and  to  then  instruct  the  computer  to  merge  the  two  targets. 
Having  merged  the  targets,  the  course  and  speed  of  the  merged  target 
is  read  directly  and  necessary  course  changes  to  avoid  or  to  intercept  the 
target  are  determined. 

Deassignment  is  assumed  to  occur  when  a target  passes  out  of  a 
convergent  zone.  When  this  occurs,  the  SQR-19  operator  manually  de- 
assigns  the  cursor.  Similarly,  the  supervisor  makes  a log  entry  indicat- 
ing the  time  and  location  at  which  the  target  exited  the  zone. 

Table  3 presents  the  number  of  different  events  (tasks)  assigned  to 
each  operator  and  the  supervisor  in  the  simulation.  The  supervisor  had  the 
heaviest  number  of  tasks  (46  tasks)  while  the  SQS-26  operator  had  the  least 
(12  tasks).  Intervals  (divisible  by  50)  between  the  assigned  tasks  and  the 
next  task  number  were  left  unassigned  in  order  to  allow  easier  identifica- 
tion between  operators. 


Table  3 


Sequenced  Event  Allocation 


OPERATOR 


TASK 

NUMBERS 


NUMBER  OF 
TASKS 


Supervisor 
AN/SQS-26  Operator 
LAMPS  Operator 
AN/SQR-19  Operator 

TOTAL 


1-46 

46 

51-62 

12 

101-126 

26 

151-178 

28 

112 


Event  Type  Data 

For  the  purposes  of  the  present  simulation,  23  event  types  were 
identified  and  quantified.  The  resulting  data  as  organized  for  model  input 
are  shown  in  Table  4.  The  column  headings  identify  the  data  type.  Three 
lines  are  used  to  describe  each  event  type.  The  first  line  gives  the  type 
number,  an  identifier  (i.e.,  prose  description),  the  average  and  standard 
deviation  of  the  duration  (in  hours)  of  the  event,  the  class  (a  model  book- 
keeping identifier),  the  number  of  pieces  of  equipment  required  for  per- 
forming the  event,  and  which  equipment.  For  example,  event  type  1 is 
concerned  with  "WATCH  SET  UP,  " has  a mean  duration  of  . 333  hours,  is 
of  class  1,  and  involves  three  units  of  equipment:  equipments  1,  2,  and  3 
which  are  the  three  separate  consoles. 

The  second  data  row  for  each  event  type  contains  the  event 
essentially,  the  mental  load  imposed  by  the  event  on  the  operators,  the 
event  kind,  the  training  code,  the  events  hazard  level,  the  number  of  men 
required  for  performing  the  event,  and  the  consumable  expenditure  rate 
in  units  per  hour.  In  the  case  of  event  type  1,  there  is  an  essentiality 
of  100  (maximum),  a mental  load  of  7 (scaled  0 to  9),  an  event  kind  of  2 
(kind  1 is  end  time  fixed  while  kind  2 is  variable  end  time,  according  to 
circumstances),  a training  code  of  1 ( 2 is  a training  event  while  1 is 
normal  event),  and  a hazard  level  of  2 (code:  1-3=  low,  4-6=  medium, 
7-9=  heavy).  Up  to  10  types  of  men  may  be  specified  as  required  for  per- 
forming different  events.  In  event  type  1,  only  one  of  man  type  1 is 
required.  No  consumables  are  considered  to  be  used  in  this  event  type. 
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AMBIGUITY  REDUCTION  0.133  0.033 


NCT  LOGS#  MODE  UtACOJIKE  O.OSJ  0.051 
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The  third  line  of  event  type  data  (Table  4)  contains  the  consumable 
expenditure  rate  in  units  and  the  expected  energy  consumption  by  each 
man  type  during  event  performance.  In  the  current  simulation  no  consum- 
ables were  assumed  to  be  used  and  the  energy  expenditure  rate  was  set  at 
100  calories  per  hour. 

Event  types  1 to  17  in  Table  4 are  operational  and  target  processing 
tasks  while  event  types  18  to  2*3  are  repair  events. 


Equipment  and  Repair  Event  Data 

Table  5 presents  the  equipment  repair  input  data  used  in  the  simula- 
tion. Repair  data  were  inserted  for  three  pieces  of  equipment,  i.e. , the 
SQS-26,  LAMPS,  and  SQR-19  consoles.  The  first  column  for  each  event 
identifies  the  event  type  to  be  used  (as  described  previously)  in  the  event 
type  data.  Other  critical  information  included  in  this  section  includes  the 
equipment  reliability  (e.g. , mean  time  between  failures  of  31.2  days  for 
equipment  number  1)  and  the  maximum  repair  time  ("duration  target"  of 
4.00).  The  data  are  used  before  each  iteration  to  determine  stochastically 
the  occurrence  or  non  occurrence  of  a failure  and  when  it  will  occur.  The 
equipment  reliability  values  were  taken  from  system  design  requirements. 
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Repair  Event  Input  Data 


Scheduled  Event  Data 


The  sequence  of  events  shown  in  Figure  2,  is  presented  in  the 
form  required  for  computer  input  in  Appendix  A.  The  types  referred  to 
in  column  2 of  Appendix  A refer  to  the  event  types  described  earlier.  The 
next  two  columns  describe  consumable  data.  No  consumable  use  was 
involved  in  this  simulation.  The  next  six  columns  of  Appendix  A refer  to 
the  task  to  be  performed  after  the  listed  task.  For  example,  in  the  case 
of  event  1,  the  next  event  to  be  performed  is  event  151.  Reference  to 
Figure  2 shows  that  this  is  the  "DETECT  A"  event  performed  by  the  SQR- 
19  operator. 

Since  sequenced  event  151  always  follows  event  number  1,  all  three 
possible  entries  are  filled  with  the  number  151,  The  probability  of  per- 
forming each  next  event  follows  the  event.  In  this  case,  a probability  of 
1.  0 (certainty)  is  used  following  the  first  probability.  Since  the  events 
are  always  considered  in  the  order  1,  2,  then  3,  the  first  probability  must 
is  indicated  as  1.0.  Following  the  next  event  data,  any  necessary  prece- 
dent event  (event  which  must  be  completed  before  present  event  can  start) 
be  indicated.  While  event  1 has  no  precedent  event,  event  2 has  a precedent 
of  event  101.  This  means  that  event  number  101  must  be  completed  before 
event  number  2 can  be  performed.  Following  the  precedent  data,  the 
temporal  requirements  are  given.  The  first  number,  identified  as  "START 
TIME,  " indicates  the  earliest  time  at  which  the  event  may  be  started.  In 
the  case  of  event  1,  the  entry  is  zero.  This  indicates  that  event  1 may  start 
at  the  start  of  the  simulation.  In  the  case  of  event  2,  an  earliest  starting 
time  of  . 17  hours  is  specified.  This  means  that  event  2 may  not  start,  or 
must  be  delayed  until  . 17  hours  into  the  mission.  The  "TIME  LIMIT" 
number  following  the  start  time  is  the  time  by  which  the  event  must  be 
finished.  If  the  task  is  not  completed  before  this  time,  it  is  terminal  1 at 
the  time  limit.  If  the  event  was  not  started  before  the  time  limit,  t!  event 
is  ignored.  The  final  entry  shown  in  Appendix  A is  the  "REPEAT/ YOUCHUP 
CODE.  " The  key  to  this  code  is  shown  in  the  column  headings.  An  entry 
of  1 indicates  an  event  which  is  repeated  in  full  if  it  is  failed  by  the  simulated 
operator(s).  An  entry  of  2 indicates  an  event  which  requires  only  a touchup 
( . 1 of  normal  event  time)  if  it  is  failed,  A 3 indicates  an  event  which 
either  allows  or  requires  no  further  action  when  it  is  failed. 


Pasic  Parameters  of  Current  Simulation 

Table  6 presents  the  baseline  parameters  used  in  the  present 
evaluation.  The  values  shown  in  Table  5 were  held  constant  across  all 
runs  except  where  specifically  indicated  in  subsequent  sections  of  this 
report.  The  nominal  pace,  or  speed,  of  all  crewman  was  set  at  1.00 
(average);  the  nominal  stress  threshold  was  set  at  2.  3 in  the  present 
evaluation  (about  average);  the  nominal  level  of  aspiration  was  1. 00  (a 
value  indicating  a desire  to  perform  in  accordance  with  proficiency)  and  so 
on.  All  of  these  values  are  used  as  means  and  actual  values  used  in  any 
specific  simulation  run  are  stochastically  determined  within  the  logic  of 
the  SW4-20  model. 

The  variables  in  the  second  group  (starting  with  "HOURS  SINCE 
LAST  SLEEP"  were  used  as  the  initial  values  in  the  simulation.  Note 
that  the  amount  of  working  time  allowed  was  four  hours. 

All  consumable  information,  initial  and  usage  rate,  were  set  at 
zero.  Accordingly,  consumables  were  not  involved  in  the  present  validation. 

Since  seasickness  was  not  a variable  of  interest  in  this  simula- 
tion, the  probabilities  of  sea  state  were  set  to  a value  of  2 to  produce  a 
continuous  mild  sea  state. 

Personnel  characteristics  included  a mean  body  weight  of  170 
pounds  with  a standard  deviation  of  15  pounds.  All  personnel  were  set  to 
be  100  per  cent  fully  qualified  in  their  primary  specialty  and  in  their 
secondary  specialty.  The  final  values  of  interest  in  this  section  are  the 
"CREW  ASSIGNMENT  TO  SHIFTS  BY  MAN.  " All  crew  members  were 
assigned  to  the  first  shift  in  the  present  simulations  since  only  one 
shift  (watch)  was  simulated. 

These  basic  parameters  were  held  constant  while  other  parameters 
were  varied  across  simulation  runs  to  allow  test  of  the  effects  of  varying 
certain  parameters  on  system  performance. 
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Baseline  (Nominal)  Parameters  in  Present  Simulation 


Q 


OUTPUT  RECORDING  OPTIONS 


| Parameters  Varied 

The  parameters  varied  in  the  simulation  were:  pace,  aspiration, 
leader  expectation,  and  per  cent  qualified.  Table  7 lists  the  levels  of  these 
variables  which  were  used  in  various  simulation  runs.  In  the  cases  of 
pace  and  aspiration,  two  additional  levels  were  selected  to  allow  considera- 
I tion  of  the  effect  of  additional  training  and/or  heightened  motivation  on 

mission  performance. 

In  the  case  of  leader  expectation,  both  higher  and  lower  levels  of 
expectation  were  employed  to  determine  the  effect  supervisor  performance 
demand  on  crew  performance.  Here,  a value  of  1.  0 indicates  that  the  simu- 
t lated  supervisor  expects  the  crew  to  perform  to  the  level  of  their  present 

capability. 

In  the  baseline  condition,  all  men  were  fully  qualified  in  their 
specialty.  One  additional  run  was  completed  with  a lower  percentage 
qualified--80  per  cent  fully  qualified  and  20  per  cent  marginally  quali- 
® fied--to  depict  the  conditions  which  might  sometime  be  found  aboard 

ship. 


Table  7 

Simulation  Parameters  Varied  in  Each  Run 

RUN 

NO.  CONDITIONS  CHANGED 

1 None  (Baseline)* 

2 PACE=  .80 

3 PACE3  . 90 

4 ASPIRATION=  .90 

5 ASPIRATION3  .95 

6 LEADER'S  EXPECTATIONS3  .85 

7 LEADER'S  EXPECTATIONS3  1.05 

8 80  per  cent  qualified  in  specialty 
20  per  cent  qualified  in  specialty 

*Unless  otherwise  specified: 

PACE=  1.00 

ASPIRATJON=  1.00 

LEADER'S  EXPECTATIONS * .95 

100  per  cent  fully  qualified  in  specialty 


CHAPTER  III 
RESULTS 


The  results  of  the  computer  simulation  of  the  mission  described 
in  Chapter  II  are  presented  relative  to  a variety  of  the  output  measures 
yielded  by  the  SW4-20  model. 


Human  Reliability 

Human  reliability  is  defined  within  the  model  as  a function  of  the 
number  of  failures  of  events  by  the  simulated  team  members.  The 
results  of  the  various  simulation  runs  relative  to  this  measure  are  pre- 
sented in  Table  8.  The  human  reliability  for  the  baseline  conditions 
was  . 58.  The  obtained  human  reliability  varied  between  . 273  and  . 993. 
Generally,  the  human  reliability  values  tend  to  be  low.  The  exceptions 
are  the  crews  with  a low  level  of  aspiration  (a  willingness  to  do  poor 
work)  and  the  crews  with  supervisors  who  possess  a low  performance  ex- 
pectation (a  willingness  to  accept  poor  work).  Increasing  crew  member 
working  speed  did  not  raise  the  obtained  human  reliability  values  to  a 
value  which  might  be  considered  acceptable  for  an  advanced  system  and, 
when  the  team  was  80  per  cent  qualified,  the  human  reliability  fell  to  . 46. 

The  human  reliability  values  were  consistently  lower  than  the 
corresponding  model  calculated  equipment  values.  This  result  suggests 
that  system  reliability  is  negatively  influenced  by  the  crew  member  un- 
reliability. This  influence  is  reflected  in  the  model  calculated  system  re- 
liability values  given  in  the  third  column  of  Table  8.  Generally,  the  sys- 
tem reliability  values  are  at  a unfavorable  level  and  can  probably  most 
easily  be  moved  upward  by  increasing  the  operator  success  rate. 

A separate  human  reliability  analysis  was  conducted,  by  operator, 
for  the  baseline  run.  The  attempt  of  this  analysis  was  to  ascertain  which 
operator(s)  contributed  most  to  the  depressed  human  reliability  values. 
The  obtained  human  reliability  values  for  the  baseline  run  are  presented 
by  operator  in  Table  9.  The  supervisor  who  is  involved  in  most  of  the 
decision  making  was  indicated  to  possess  the  lowest  human  reliability. 
This  indication  suggests  the  need  for  decision  aiding.  While  the  decision 
events  are  not  necessarily  time  consuming,  failure  of  such  events  im- 
pedes performance.  Prosecution  of  the  events  of  the  mission  by  other 
sonar  system  team  members  cannot  take  place  until  after  the  supervisor 
has  made  the  appropriate  decision. 


Table  8 


Human, 

Equipment,  and  System  Reliability  Under 
Various  Simulated  Conditions 

ReZiablLUy 

SIMULATION  RUN 

huMan 

Baseline* 

.58 

1.00 

.58 

Pace 

.8 

.62 

1.00 

.62 

.9 

.61 

.85 

.46 

1.0* 

.58 

1.00 

.58 

Aspiration 

.90 

.84 

.90 

.74 

.95 

.61 

.95 

.60 

1.00* 

.58 

1.00 

.58 

Leader  Expectation 

.85 

.99 

1.00 

.99 

.95* 

.58 

1.00 

.58 

1.05 

.27 

.90 

.17 

% Fully  Qualified 
80-20 

.46 

.90 

.36 

100-0* 

.58 

1.00 

.58 

* Baseline  run 
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Table  9 


iabi 1 i +y  for 

Each  Simu la tod  Oper. 

OPERATOR 

Hum  RELIABILITY 

Supervisor 

.49 

SQS-26 

.84 

LAMPS 

.58 

SQR-19 

.63 

t Individual  analyses  were  also  completed  in  terms  of  first  trial 

success  as  a function  of  operator  speed  (pace)  relative  to  three  selected 
critical  events.  The  events  selected  for  detailed  examination  were: 
target  detection,  target  classification,  and  target  reacquisition.  The 
operator  paces  used  in  these  runs  were  1.0  (average),  .9  (faster  than 
average),  and  . 8 (much  faster  than  average).  A simulated  sonar  team  with 
t a pace  of  . 8 works  about  20  per  cent  faster  than  a team  with  a pace  value 

of  1.0,  other  things  being  equal.  Within  the  model,  pace  does  not  directly 
affect  success  probability.  Rather,  pace  affects  success  indirectly, 
through  the  effect  of  pace  on  aspiration  and  stress.  The  results  of  these 
analyses  are  presented  in  Figures  3,4,  and  5. 

^ Depressed  human  reliability  values  are  similarly  indicated  for  the 

events  selected  for  individual  analysis  and  the  effect  of  the  pace  para- 
meter was  greater  for  these  individual  tasks  than  for.the  overall  set  of 
tasks.  This  would  suggest  that  increased  human  reliability  on  the  overall 
is  required  rather  than  emphasis  on  individual  events. 

^ The  three  points  in  each  of  Figures  3,  4,  and  5 were  used  to  calcu- 

late, by  the  least  squares  method,  the  theoretical  ideal  fit.  The  result- 
ing line  of  best  fit.  is  shown.  Also  shown  is  the  correlation  (r)  between 
operator  pace  and  percentage  of  first  trial  successes.  The  correlation 
may  be  used  as  a rough  quantification  of  the  relationship.  The  equation 
describes  the  theoretical  line  of  best  fit  where  Y represents  the  ordinate 
and  X represents  the  abscissa. 
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PERCENT  OF  FIRST  TRIAL  SUCCESSES  IN  TARGET  DETECTION 
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FIGURE  3.  PERCENT  OF  FIRST  TRIAL  SUCCESSES  ON  TARGET 
DETECTION  AS  A FUNCTION  OF  OPERATOR  PACE  . 
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FIGURE  4.  PERCENT  OF  FIRST  TRIAL  SUCCESSES  ON  TARGET  CLASSIFICATION 
AS  A FUNCTION  OF  OPERATOR  PACE . 


Human  Availability 


Within  the  SW4-20  model  human  availability  is  defined  as  a 
function  of  time  that  there  is  work  to  be  performed  but  during  which  the 
work  can  not  be  accomplished  because  the  crew  members  are  perform- 
ing other  work  (i.e.,  unavailable).  The  obtained  availability  values,  as 
presented  in  Table  10,  tend  to  be  adequate  although  not  high.  This  suggests 
that  the  manning  level  involved  in  the  simulation  was  sufficient.  When 
viewed  in  coordination  with  the  prior  reliability  data,  these  results  suggest 
combined  SQS-26,  LAMPS,  SQR-19  system  does  not  require  more  men 
but,  rather,  it  requires  easier  performance. 

The  human  availability  indices  tend  to  be  lower  than  the  equip- 
ment availability  indices  and,  this  finding  suggests  further  support  for  the 
prior  contention  relative  to  the  need  for  improved  design  from  the 
operator  point-of-view. 

As  for  the  human  reliability  analyses,  the  availability  during  the 
baseline  run  was  calculated  separately  by  operator.  The  results  of 
this  analysis  are  presented  in  Table  11. 

As  for  the  human  reliability,  the  supervisor  was  indicated  to  re- 
present the  weakest  human  component  in  the  system.  The  possibility 
exists  that  work  aids/decision  aids,  as  suggested  above,  would  tend  to 
lessen  the  decision  load  on  the  supervisor  and  make  him  more  available. 
Alternatively,  workload  redistribution  may  be  indicated.  We  also  note 
that  much  of  the  communication  load  is  carried  by  the  supervisor. 

Possibly,  a communications  analysis  would  yield  a basis  for  lightening 
the  load  on  the  supervisor.  Certainly,  a .44  availability  value  for  the 
supervisor  is  unacceptably  low  and  casts  doubt  on  how  well  the  combined 
SQR-26,  LAMPS,  SQS-19  system  is  integrated. 


Table  10 


Human  and  Equipment  Availability  Under  Various  Simulated  Conditions 


SIMULATION  RUN 

HUMAN  AVAILABILITY 

EQUIPMENT  AVAILABILITY 

Baseline* 

.87 

1.00 

Pace 

.80 

.91 

1.00 

.90 

.79 

.97 

1.00* 

.81 

1.00 

Aspiration 

.90 

.81 

.97 

.95 

.85 

.99 

1.00* 

.81 

1.00 

Leader  Expectation 

.85 

.80 

1.00 

.95* 

.81 

1.00 

1.05 

.81 

.98 

% Fully  Qualified 

80-20 

.72 

.98 

100-0* 

.81 

1.00 

*Ba8eiine  run 
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Human  Mean  lime  to  Repair 


Human  mean  time  to  repair  reflects  the  amount  of  time  consumed 
by  an  equipment  operator  or  maintainer  when  he  repeats  or  "touches  up" 
a failed  event.  In  view  of  the  rather  low  obtained  human  reliability  in- 
dices, one  would  anticipate  that  the  predicted  human  mean  time  to  repair 
values  would  be  high.  The  human  mean  time  to  repair,  as  indicated  by 
the  computer  model  for  the  various  conditions  simulated,  is  presented 
in  Table  12.  For  the  baseline  run,  the  human  mean  time  to  repair  was  in- 
dicated to  be  .06  hours  or  an  average  of  3.6  minutes  for  each  failed  event. 
In  view  of  the  rather  high  number  of  failed  events,  it  becomes  apparent 
that  the  total  human  availability  would  increase  markedly  if  the  design  of 
the  combined  SQS-26,  LAMPS,  SQR-19  system  was  reconsidered  so  as  to 
provide  fewer  human  failures  and,  accordingly,  less  total  human  repair 
time. 


Table  3 indicated  a total  of  112  events  across  all  sonar  team 
members  for  the  simulated  mission.  Assuming:  (1)  an  average  of  28 
events  per  team  member  (112  events/4  team  members=  28  events  per 
team  member  and  (2)  a human  reliability  of  .58  (baseline  results),  then 
the  simulation  suggests  that,  on  the  average,  each  simulated  team  member 
failed  11.  8 events  (unreliability  was  100  -.58=  , 42:,42  x 28=  11.76).  With 
a human  mean  time  to  repair  value  of  3.  6 minutes,  this  suggests  that  about 
42.  3 minutes  (3.  6 x 11.  8=  42.  3)  of  the  four  hour  watch  were  spent  in  re- 
peating or  "touching  up"  failed  events. 

Table  12  also  indicates  that  the  human  MTTR  tended  to  be  higher 
than  the  equipment  MTTR.  Accordingly,  the  result  is  that  system  MTTR 
is  depressed. 
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Table  11 


Human  Availability  for  Each  Simulated  Operator  (baseline  Run) 

OPERATOR  HUMAN  Al/AI  LABILITY 


Supervisor  . 44 
SQS-26  .86 
LAMPS  . 98 
SQR-19  . 93 


Table  12 


System, 

Human,  and  Equipment  MTTR  (Hours) 
Various  Simulated  Conditions 

under 

MTTR 

SIMULATION  RUN 

Baseline* 

.06 

.00 

.06 

Pace 

80 

.03 

.00 

.03 

.90 

.99 

.01 

.12 

1.00* 

.06 

.00 

.06 

Aspiration 

.90 

.05 

.30 

.07 

.95 

.04 

.10 

.05 

1.00* 

.06 

.00 

.06 

Leader  Expectation 

.85 

- 

.00 

- 

.95* 

.06 

.00 

.06 

1.05 

- 

.10 

- 

% Fully  Qualified 

80-20 

.27 

.10 

.12 

100-0* 

.06 

.00 

.06 

* Baseline  run 
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The  human  MTTR  value  for  each  simulated  team  member  is  pre- 
sented in  Table  13.  As  indicated  in  Table  13  the  SQS-26  operator  seems  to 
have  contributed  most  to  elevating  the  human  mean  time  to  repair  values. 
Evidently,  the  events  failed  by  the  SQS-26  operator  require  long  repetition 
and  touchup  times --even  though  the  reliability  and  availability  of  this 
operator  were  not  the  lowest  noted.  The  supervisor,  who  was  previously 
indicated  to  possess  depressed  reliability  and  availability  was  indicated 
to  possess  the  lowest  human  mean  time  to  repair.  This  finding  suggests  that 
the  problem  involved  in  the  supervisory  events  is  not  that  the  individual 
events  are  too  time  consuming.  Rather,  initial  failure,  due  to  task 
difficulty  seems  to  be  the  problem. 

Table  13 

Human  MTTR  for  Each  Simulated  Operator 

OPERATOR  HUMAN  MTTR 


Supervisor 

.11 

SQS-26 

.35 

LAMPS 

.14 

SQR-19 

.15 

Performance  Adequacy 

The  model  also  calculates  a performance  adequacy  value  which 
aims  to  provide  an  index  of  how  well  the  simulated  crew  performs  the 
various  events  of  the  mission.  The  obtained  index  can  vary  from  zero  to 
100  and  is  based  on  the  current  competence  of  the  crew  member(s)  per- 
forming each  event,  the  stress  level  at  the  time  of  event  performance, 
the  current  physical  status  of  the  crew  member(s)  performing  the  event, 
and  the  current  aspiration  level(s).  The  obtained  overall  performance 
adequacy  for  the  baseline  condition  is  presented  in  Table  14  along  with 
the  performance  adequacy  for  each  of  three  selected  events.  The  values 
shown  in  Table  14  suggest  moderate  performance  adequacy  for  each  of 
the  events  selected  for  individual  examination  as  well  as  for  all  of  the 
tasks  within  the  mission  simulated.  From  the  point  of  view  of  this  analy- 
sis. the  classification  event  would  benefit  most  from  increased  design 
emphasis.  The  depressed  performance  adequacy  values  are  a function 
of  the  time  stress  on  the  operators  because  the  baseline  condition  included 
a 100  per  cent  fully  qualified  crew  with  an  aspiration  of  1.0  and  there 
was  little,  if  any,  physical  degradation  in  the  simulation  as  performed. 
These  values,  when  coupled  with  the  human  reliability  and  availability 
data,  presented  earlier,  lend  further  support  to  contentions  favoring  a 
need  for  operability  emphasis  in  the  design  of  the  combined  SQS-26, 
LAMPS,  SQR-19  systems. 
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Table' 14 


Overall  Performance  Adequacy  and  Performance  Adequacy 
on  Three  Selected  Evenfs  (Baseline  Conditions) 


EVENT  PERFORMANCE  AVEQliACV 


Overall 

.62 

Target  Detection 

.87 

Classification 

.73 

Reacquisition 

.77 

Leader  Expectation 


The  leader  expectation  focuses  on  the  quality  of  performance 
demanded  and  acceptable  to  the  simulated  supervisor  from  the  three 
simulated  operators. 

In  the  baseline  simulation,  an  expectation  of  . 95  was  employed. 

This  value  was  bracketed  by  leader  expectations  of  . 85  and  1.  05  in  other 
simulations. 

As  shown  in  Figures  6 and  7,  performance  adequacy  and  human 
reliability  decreased  as  leader  expectation  increased.  That  is,  only 
when  leader  expectation  was  low  (i.  e.  .willingness  to  accept  inferior  work) 
was  performance  adequacy  judged  reasonably  good.  As  leader's  expectations 
increased  to  the  high  quality  work  required  by  this  mission,  performance 
adequacy  was  shown  to  decrease. 

Percentage  of  Crew  Fully  Qualified 

Two  levels  of  crew  qualification  were  simulated.  The  baseline  and  all 
of  the  other  runs  except  one,  assumed  that  all  personnel  were  100  per  cent 
fully  qualified  in  their  primary  specialty,  sonar  operation  and  maintenance. 

A single  run  was  made  under  the  condition  of  80  per  cent  of  the  crew  fully 
qualified  and  20  per  cent  minimally  qualified. 

Figure  8 presents  the  resulting  difference,  as  indicated  by  the 
simulation,  between  the  two  qualification  levels  in  overall  percentage  of 
first  trial  success.  Figure  8 shows  a 41  per  cent  drop  in  the  mean 
number  of  first  trial  successes  as  a result  of  reducing  the  qualification 
level.  Similarly,  Figure  9 indicates  that  the  mean  percentage  of  tasks 
failed  increased  as  the  qualification  index  was  lowered.  The  overall 
trends  of  human  reliability,  as  shown  in  Figure  10  and  11,  were  also  in- 
dicated by  the  simulation  model  to  suffer  decrement  when  the  qualifica- 
tion level  was  reduced.  These  data  suggest  that  performance  on  the 
combined  SQS-26,  LAMPS,  SQR-19  system  hinges  on  a well  trained  crew. 

A discussion  of  the  ability  of  the  Training  Command  to  meet  stringent 
training  requirements  is  beyond  the  scope  of  the  present  report,  However, 
it  seems  that  this  training  requirement  should  be  held  in  mind  relative 
to  the  design  of  the  system  in  question. 
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FIGURE  7.  HUMAN  RELIABILITY  AS  AFFECTED  BY 
LEADER  EXPECTATIONS  . 
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80  100 
PERCENT  FULLY  QUALIFIED 

FIGURE  8.  OVERALL  PERCENT  OF  FIRST  TRIAL  SUCCESSES  AS  A 
FUNCTION  OF  PERCENTAGE  OF  CREW  FULLY  QUALIFIED. 


80  100 
PERCENT  FULLY  QUALIFIED 

FIGURE  9.  OVERALL  PERCENT  OF  TASKS  FAILED  AS  A FUNCTION  OF 
PERCENTAGE  OF  CREW  FULLY  QUALIFIED. 
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FIGURE  11.  HUMAN  AVAILABILITY  AS  AFFECTED  BY  PERCENT  QUALIFIED. 
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CHAPTER  IV 


SUMMARY  AND  DISCUSSION 


The  purpose  of  the  present  work  was  to  provide  an  estimate  of  the 
merit  of  the  combined  SQS-26,  LAMPS,  SQR-19  system  from  the  point 
of  view  of  the  adequacy  of  the  design  vis-a-vis  human  performance.  To 
this  end,  a previously  developed  and  validated  computer  simulation  model 
was  employed  to  simulate  the  performance  of  the  sonar  team  during  a 
watch  which  involves  processing  five  targets  and  eventually  attacking  one  of 
the  targets.  The  situation  was  known,  at  the  outset,  to  impose  a heavy 
workload  on  the  simulation  team. 

Simulation  of  the  same  mission  was  performed  for  various  operator 
skill  mizes,  work  speeds,  levels  of  aspiration,  and  expected  levels  of 
performance.  The  results  indicated  the  predicted  human  performance  to 
be  at  a level  which  would  ordinarily  be  considered  unacceptable  in  an 
advanced  system.  The  human  reliability  (baseline  condition),  on  the  over- 
all, was  predicted  to  be  .58.  The  baseline  overall  human  availability  and 
mean  time  to  repair  values  were  respectively  . 81  and  .06  hours.  These 
values  were  also  held  to  possess  negative  implications  relative  to  the 
anticipated  human  performance  in  the  integrated  system.  Human  relia- 
bility, as  measured  here,  is  essentially  an  index  of  the  number  of  mission 
events  failed.  Human  availability  indexes  time  during  which  required 
work  is  put  aside  because  the  team  members  are  performing  other  work. 
Human  mean  time  to  repair  indexes  time  involved  in  repeating  or  touch- 
ing up  previously  failed  events. 

Because  the  human  performance  indices  were  lower  than  the 
corresponding  equipment  oriented  indices,  the  simulation  indicated  sys- 
tem oriented  indices  (reliability,  availability,  maintainability)  also 
tended  to  be  depressed. 

Analysis  relative  to  the  individual  operators,  indicated  the  team 
supervisor  to  possess  the  lowest  reliability  and  availability  while  the 
SQS-26  operator  was  indicated  to  possess  the  highest  mean  time  to  re- 
pair. 


Across  team  members,  the  baseline  number  of  successes  on  the  first 
trial  was  53  per  cent.  Increased  crew  speeds  and  motivational  levels 
were  explored  in  order  to  estimate  how  much  this  performance  might  be 
improved.  They  were  found  to  be  of  limited  value.  An  overall  20  per 
cent  speeded  up  crew  increased  the  percentage  of  first  trial  successes 
only  to  57  per  cent.  A reduction  in  leader’s  expectation  increased  the 
first  trial  successes  to  83  per  cent.  However,  this  value  includes  a 67 
per  cent  success  rate  for  the  critical  target  reacquisition  events. 

As  is  true  of  any  simulation  model,  a model's  output  is  only  as 
good  as  the  input  employed.  In  the  present  simulations,  the  mission 
scenario  input  and  associated  data  were  prepared  in  coordination  with 
personnel  of  the  Naval  Underwater  Systems  Center.  Accordingly,  some 
confidence  may  be  placed  in  the  inout  aspects  of  the  simulation.  If  the 
scenario  had  contained  fewer  targets  to  be  classified  or  more  time  avail- 
able in  which  to  process  targets,  a considerably  better  impression  of  the 
system's  capability  might  have  been  obtained.  Further  simulations  might 
be  indicated  for  situations  involving  lighter  target  configurations. 

The  discussion  within  the  report  emphasized  that  human  perform- 
ance improvement  in  the  combined  system  seems  to  be  associated  with 
improving  the  man/ machine  interface  and  the  communications  network  as 
well  as  with  workload  lightening/ redistribution  and  job  aid  provision.  A 
considerable  training  requirement  was  also  indicated. 


Conclusions 

For  the  conditions  and  related  scenario  simulated,  the  results  of 
the  application  of  the  SW4-20  model  to  the  combined  SQS-26,  LAMPS, 
SQR-19  system  suggested: 

1.  The  human  reliability  appeared  to  be 
depressed  and  considerably  lower 
than  the  equipment  reliability. 

2.  Similarly,  the  human  availability  and 
human  mean  time  to  repair  were  depressed 
below  a level  which  would  seem  acceptable 
for  an  advanced  system. 
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3.  Of  the  four  operators  considered-- 
supervisor,  SQS-26  operator,  LAMPS 
operator,  and  SQR-19  operator--the  super- 
visor seemed  to  degrade  the  overall  human 
reliability  and  availability  most  while  the 
SQS-26  operator  seemed  to  degrade  the 
overall  human  mean  time  to  repair  the  most. 

4.  The  man/ machine  interfaces  of  the  system, 
including  the  communication  links,  seem  to 
require  design  emphasis  in  order  that  the 
operability  of  the  system  may  be  increased. 
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FOREWORD 


Hutnan  reliability  is  a necessary  consideration  when  the  contribu- 
tion of  system  operators  to  the  unreliability  of  a total  system  is  an  issue 
of  interest.  Human  reliability  deals  with  that  aspect  of  total  system  un- 
reliability attributable  to  human  error  or  human  induced  failure.  The 
human  reliability  concept  includes  all  aspects  of  system  operation  and  is 
measured  in  terms  of  demonstrated  operator/ maintainer  performance. 

These  Human  Performance  Reliability  Demonstration  Guidelines 
present  desirable  steps  and  procedures  for  employment  when  the  need 
exists  for  assuring  that  an  equipment  unit,  subsystem,  or  system  are  not 
limited  by  human  failure.  The  Guidelines  are  not  "standards"  or  "specifi- 
cations" and  in  no  way,  are  the  Guidelines  intended  to  replace,  supercede, 
or  minimize  such  documents  as  MIL-STD-471  (Maintainability  Demonstra- 
tion), MIL-STD-  72 IB  (Definitions  of  Effectiveness  Terms  for  Reliability, 
Maintainability,  Human  Factors,  and  Safety),  MIL-STD-756A  (Reliability 
Prediction),  and  the  like.  While  the  Guidelines  could  lead  to  a formal 
"standard1  or  "specification,"  practical  military  service  and  industrial 
experience  with  their  use,  implementation,  advantages,  and  associated 
problems  seems  required  before  such  finalization  can  take  place. 

In  a sense,  the  Guidelines  represent  a first  step  towards  quality 
control  and  assurance  from  the  point -of-view  of  the  interface  of  the 
human  and  the  equipment  units  in  an  equipment  unit/ subsystem/ system. 
Recent  experience  has  indicated  this  interface  to  constitute  one  of  the 
weakest,  if  not  the  weakest,  design  links.  Quite  often,  such  softness  in 
a design  is  compensated  for  by  increased  training  requirements,  heightened 
manning,  or  by  using  higher  rated  operators/ maintainers.  However, 
such  fixes  are  costly  and  not  always  possible.  For  example,  recent  in- 
formation indicated  manpower  costs  to  represent  55  per  cent  of  the  total 
life -cycle  cost  of  a modern  ship. 

The  development  of  the  Guidelines  was  iterative  in  nature  and 
considerable  reliance  was  placed  on  the  advice  and  opinions  of  knowledge- 
able persons  holding  Governmental  equipment  development  responsibility 
and  on  the  insights  of  industrial  design  and  development  personnel.  At 
the  outset,  a complete  outline  of  the  Guidelines  was  prepared.  A group 
of  industrial  and  Governmental  representatives  was  called  together  to 
critique  the  outline  with  specific  reference  to  the  following  questions: 

•Can  you  live  with  such  a set  of  standards? 

•What  part(s)  can't  you  live  with? 

•What  should  be  changed  and  how? 


•What  are  the  cost  impacts? 


•How  will  your  management  react  to  various 
aspects  of  the  standards? 

•What  is  included  which  should  be  deleted? 

•What  included  material  should  be  emphasized 
more?  Deemphasized? 

•How  should  the  order  of  the  parts  be  modified? 

•What  included  content  material  is  useless?  Wrong? 

•What  material  is  impractical  and  how  could  it  be 
made  more  practical? 

•What  terminology  is  unclear  or  not  consistent 
with  ordinary  use? 

•What  content  may  conflict  with  policies  of  your 
organization? 

The  comments  of  these  persons  were  considered,  and  appropriate  ad- 
justments were  made  in  the  preliminary  outline.  Then,  a first  draft  was  pre- 
pared and  distributed  to  a set  of  reviewers  who,  again,  represented  the  civil 
and  the  Governmental  sectors.  The  comments  and  opinions  of  the  reviewers 
were  integrated  into  this  final  form  of  the  Guidelines.  Quite  obviously,  the  re- 
sponsibility for  the  final  product  is  that  of  Applied  Psychological  Services  alone. 

In  developing  the  Guidelines,  there  was  a deliberate  attempt  to  employ 
terminology  relative  to  the  human  in  a system  which  parallels  that  applied  to 
equipment  units,  e.  g. , human  reliability,  human  mean  time  to  repair,  human 
availability.  Moreover,  there  was  an  attempt  to  define  such  terms,  relative  to 
the  human  in  the  system,  in  a manner  which  parallels  the  use  of  these  terms 
as  they  are  applied  to  the  equipment.  This  emphasizes  our  thinking  that  total 
reliability,  availability,  and  maintainability  are  functions  of  both  human  and 
equipment  considerations. 

The  human  reliability  demonstration  is  not  intended  to  establish  the 
human  reliability.  Reliability  never  is  a static  or  constant  figure.  It  is  de- 
termined at  various  points  in  the  life  cycle  of  a system  and  may  vary  over 
these  points  as  a function  of  a wide  variety  of  influences.  Similarly,  these 
Guidelines  cannot  be  considered  static.  As  experience  with  the  Guidelines 
develop,  areas  for  revision  and  update  will  certainly  develop. 

Finally,  we  note  that  human  reliability  is  a product  of  proper  design -- 
not  a consequence  of  a test  or  demonstration. 

APPLIED  PSYCHOLOGICAL  SERVICES,  INC. 

May  1977 
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HUMAN  PERFORMANCE  RELIABILITY  DEMONSTRATION  GUIDELINES 


1. 

Introduction 

> 

1.  1 

Purpose.  These  Guidelines  consider  the  test  methods  and 

procedures  which  can  be  applied  in  order  that  the  attain- 
ment of  a predicted  or  desired  level  of  human  reliability 
can  be  demonstrated. 

> 

1.2 

Scope.  This  set  of  human  performance  reliability  demonstra- 
tion guidelines  is  designed  to  be  compatible  with  any  formal 
human  engineering  and  equipment  reliability  program  which 
is  implemented  throughout  the  development  of  an  equipment/ 
subsystem/ system.  The  Guidelines  are  applicable  to  mechanical, 
electromechanical,  electronic,  and  electrical  equipment,  sub- 
systems, and  systems. 

> 

The  human  performance  reliability  (HR)  demonstration  may 
overlap  in  part  with  the  maintainability  demonstration  (MIL- 
STD-471).  These  two  demonstrations  may,  at  times,  consider 
the  same  set  of  tasks  from  different  points  of  view  (i.  e.,  with 
different  criteria).  Similarly,  the  human  performance 

> 

reliability  demonstration  may  be  integrated  with  the  maintaina- 
bility demonstration. 

1.  3 

Application.  Human  reliability  is  an  important  consideration 
whenever  attainment  of  a desired  level  of  integrated  system 
reliability  is  to  be  demonstrated.  As  such,  human  reliability 

1 

is  a part  of  a total  program  and  not  a substitute  for  a human 
factors,  reliability,  or  quality  assurance  program. 

1.4 

Relationship  to  Equipment  and  Integrated  System  Reliability. 
The  various  aspects  of  human  reliability  are  viewed  in  a 
fashion  which  parallels  the  treatment  of  equipment  relia- 

1 

bility  and  integrated  system  reliability.  With  respect  to 
each,  the  following  measures  provide  unique  information: 
reliability,  availability,  mean  time  to  repair  (MTTR),  and 
mean  time  between  failures  (MTBF).  The  interpretation  of 

t 

these  terms  with  respect  to  human  performance  is: 

« 

m 

% 

i I 
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a.  Human  reliability  is  the  probability  that  the 
humans  in  a system  will  be  able  to  c omplete 
required  performance  without  error  and  with- 
in time  constraints. 

b.  Human  availability  is  the  proportion  of  assigned 
time  that  human  operators  and/or  maintainers 
are  available  to  conduct  required  tasks. 

c.  Human  MTTR  is  a measure  of  the  mean  time 
required  for  an  operator  to  correct  and  ac- 
curately execute  a previously  failed  subtask. 

Terms  and  Definitions: 

ASVAB  (Armed  Services  Vocational  Aptitude  Battery) 
A paper  and  pencil  test,  designed  and  developed  by 
the  armed  forces,  for  assessing  aptitude  for  a wide 
variety  of  civil  and  military  jobs. 

Criticality:  The  extent  to  which  failure  to  perform 
properly  a given  subtask,  subtask  sequence,  or  task 
will  degrade  attainment  of  mission  objectives  or  will 
degrade  system  performance. 

Empirical:  Related  to  facts  or  data  as  opposed  to 
theory  and  speculation. 

Fix-up:  Effort  expended  in  redoing  a part  of  a task 
which  was  not  successfully  completed. 

Frequency:  A measure  of  how  often  a given  opera- 
tional or  maintenance  task  or  subtask  occurs  during 
actual  equipment,  subsystem,  or  system  employ- 
ment. 

Frequency-Criticality  Matrix:  An  integrated  repre- 
sentation of  the  frequency  of  occurrence  and  of  the 
criticality  of  each  task  performed  by  operators. 

Hard  Copy:  A product  of  job  or  test  performance 
which  may  be  repeatedly  and  independently  scored 
and  measured. 
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HR  Demonstration;  Moc  k operational  test  of  the 
ability  of  anticipated  operational  personnel,  or 
their  equivalents,  to  function  without  failure  in 
their  anticipated  role.  This  demonstration  may 
be  preceded  by  any  number  of  developmental  HR 
tests  at  any  stage  in  the  development  of  the 
equipment/subsystem/ system. 

Norm:  A single  value  or  a range  of  values  constitut- 
ing the  usual  performance  of  a given  group. 

Operators:  The  technicians  operating  an  equipment/ 
subsystem/ system  during  the  HR  demonstration. 

Percentile  Equivalent:  A score  representing  the  pro- 
portion of  persons  in  a given  sample  who  fall  below  a 
given  raw  score. 

Reliability  of  Measurement:  The  consistency  or 
repeatability  of  a performance  measure.  One 
particularly  relevant  form  of  reliability  is  termed 
interrater  reliability  which  refers  to  the  agreement 
between  various  judges  of  the  same  performance. 

Subtask:  A logical  unit  of  performance  which  is  not 
logically  reduceable  to  a smaller,  observable  unit. 

Task:  A logical  group  of  actions  which  form  a 
coherent  set  of  activities.  Tasks  are  more  global 
in  scope  than  subtasks  as  customarily  included  in 
a task  analysis.  For  example,  an  equipment  align- 
ment and  calibration  process  which  requires  40 
minutes  and  involves  many  interactive  steps  may 
be  defined  as  one  task. 

Examples  of  tasks  are: 

making  a parts  replacement 

operating  an  equipment  under  a specific 
set  of  circumstances 


performing  corrective  maintenance 


performing  preventive  maintenance 


making  a periodic  check 

Technical  Training:  The  totality  of  planned  circum- 
stances, instructions,  and  directed  activity  contribut- 
ing to  performance  or  learning. 

Theoretical:  A general  principle,  supported  by 
considerable  data,  proposed  as  an  explanation. 

Transient  Copy:  A product  of  job  or  test  perform- 
ance which  is  momentary  in  nature,  e.g.,  a measure- 
ment which  must  be  made  during  a circumscribed 
period  of  time.  This  term  also  applies  to  a product 
which  is  destroyed  by  the  measurement  process. 

Vocational  Training:  Any  form  of  training,  whether 
given  in  school  or  elsewhere,  whose  purpose  is  to 
fit  an  individual  for  effective  pursuit  of  a recognized, 
profitable  employment. 

3. 


3.  1 


3.2 
3.2.  1 


3.2.2 


Detailed  Requirements:  --Preparation  for  Conduct  of 
Demonstration.  All  actions  should  be  completed  prior 
to  conducting  the  actual  HR  demonstration. 

Demonstration  Plan.  Prior  to  the  completion  of  any  sub- 
sequent steps,  the  contractor  should  submit  a "HR  demoh- 
stration  plan.  This  plan  should  include,  but  not  be  limited 
to:  (a)  a detailed  explanation  of  how  he  will  complete  the 
various  procedures  and  actions  indicated  throughout  these 
Guidelines,  (b)  a milestone  chart  with  anticipated  date  of 
milestone  completion,  (c)  names  of  responsible  personnel 
and  contacts,  and  (d)  management  information. 

Task/Subtask  Identification 

Identification  of  Tasks.  A list  of  tasks  should  be  developed 
which,  as  a whole,  exhaustively  describes  the  performance 
of  the  human  operator(s)  in  the  system. 

Responsibility,  Responsibility  for  the  identification  of  tasks 
should  rest  with  the  contractor. 
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Approval.  Approval  of  the  procuring  agency  should  be 
sought  for  this  partitioning  of  duties  into  a task  list  prior 
to  further  use  of  the  task  list  in  the  development  of  the 
HR  demonstration. 

Task  Frequency.  The  relative  frequency  of  execution  of 
each  task  should  be  determined.  This  determination  should 
consider  normal  operation,  emergency  operation,  and  opera 
tion  under  degraded  conditions. 

Method.  The  frequency  of  occurrence  of  each  task  may  be 
determined  using  either  empirical  or  theoretical  methods. 
Usually,  the  same  method  will  be  used  for  all  tasks  in 
the  task  list. 

For  example,  the  frequency  of  occurrence  of  each  task 
may  be  determined  by  tallying  the  number  of  occurrences 
of  that  task  during  a simulated  or  actual  application  of  the 
system.  This  application  would  include  the  equivalent  of  at 
least  one  full  day's  operation  with  both  normal,  emergency, 
and  degraded  conditions  considered. 

Or,  the  frequency  of  occurrence  of  each  task  may  be 
determined  from  an  analysis  of  the  mission  profile  of 
each  type  of  application  of  the  system  together  with  a 
determination  of  the  relative  frequency  of  occurrence  of 
each  type  of  application.  Each  task  would  be  assigned  to 
the  highest  frequency  level  indicated  by  this  analysis. 

For  example,  if  a task  is  respectively  judged  to  occur 
with  low,  low,  and  high  frequency  for  the  normal,  emer- 
gency, and  degraded  modes,  the  task  would  be  assigned 
a high  frequency. 

Levels.  Criteria  for  three  levels  of  frequency  should  be 
developed.  These  levels  should  be  termed  "low,  " "moder- 
ate, " and  "high"  frequency. 

Task  Criticality.  The  criticality  of  the  execution  of  each 
task  should  be  determined  by  developing  estimates  of  the 
effects  of  improper  performance  of  the  task  on  the  attain- 
ment of  mission  objectives  or  on  system  performance.  To 
this  end,  each  task  may  be  judged  on  a hierarchical  scale 
composed  of  three  categories --"low"  criticality,  'moderate' 
criticality,  and  "high"  criticality. 


3,  3 Sample  Selection,  Because  the  total  number  of  tasks  in- 

cluded in  the  task  list  will  be  more  than  the  number  of  tasks 
to  be  included  in  the  HR  demonstration,  a sample  of  the 
tasks  will  be  required.  The  goal  of  this  sampling  is  to 
assure  that  the  tasks  selected  for  inclusion  in  the  demonstra- 
tion are  truly  representative  of  the  population  of  tasks  in- 
cluded in  the  task  list.  Any  acceptable  sampling  scheme 
may  be  employed.  One  scheme  is  to  first  decide  on  the 
total  number  of  tasks  which  can  be  accomplished  in  the  HR 
demonstration.  It  is  anticipated  that  about  one  week  will 
ordinarily  be  allocated  to  the  HR  demonstration.  The  deci- 
sion relative  to  the  total  number  of  tasks  to  be  included  in 
the  demonstration  can  be  made  in  view  of  the  time  available 
and  the  mean  amount  of  time  required  to  organize  and 
implement  the  demonstration  of  each  task. 

Once  the  total  number  of  tasks  to  be  included  in  the  demon 
stration  is  known,  a two  stage  proportional  sample  is  drawn 
in  such  a manner  that  twice  the  number  of  tasks  to  be  included 
in  the  final  demonstration  is  selected.  Details  of  one  method 
for  this  selection  are  shown  below. 

The  first  step  involves  construction  of  a frequency- 
criticality  distribution  which  will  serve  as  the  basis  for 
the  selection  of  tasks  nominated  for  inclusion  in  the  HR 
demonstration.  This  distribution  integrates  the  frequency 
and  criticality  data  for  each  task.  Each  task  is  categorized 
into  one  cell  of  a 3 x 3 frequency-criticality  matrix.  Figure 
1 shows  how  this  matrix  should  appear.  Each  cell  entry  in 
Figure  1 contains  the  task  number  of  each  task,  identified 
and  assigned  to  a frequency  and  to  a criticality  category 
previously.  For  example,  task  26  is  low  on  both  criticality 
and  frequency;  task  23  is  a high  frequency-high  criticality 
task. 

The  second  step  involves  determining  the  number  of 
tasks  at  each  criticality  (row  total)  and  the  number  of  tasks 
at  each  frequency  level  (column  total).  The  row  totals  will 
sum  to  the  total  number  of  tasks  (50  in  this  case)  as  will 
the  column  totals. 
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FREQUENCY 


Criticality 

Low 


Moderate 

High 

Column  Total 


Low 

Moderate 

High 

1,3,26,27, 

16,17,18,32, 

4,24,25,40, 

44,45,46 

48,49 

41,42 

15,20,28, 

2,19,20,43, 

5,6,31,33 

29 

50 

8,9,10,21, 

11,12,13, 

7,14,23, 

34,35 

22,36,37,38 

39,47 

17 

18 

15 

Paw 

Total 

19 

13 


18 


50 


Figure  1.  Example  of  f requency-cr i t i ca I i t y ma t r i x. 


The  relative  proportion  of  tasks  to  be  selected  from  each 
cell  is  calculated  as  the  product  of  the  appropriate  frequency 
and  criticality  divided  by  the  sum  of  the  cross  products  (ap- 
propriate row  total  multiplied  by  column  total).  The  sum  of 
the  cross  products  should  be  calculated  to  verify  accuracy. 
However,  it  will  always  be  equal  to  the  square  of  the  total 
number  of  tasks.  In  the  example,  for  low  frequency  and 
moderate  criticality  there  is  a cross  product  of  221  (13  x 17) 
and  the  cross  product  sum  is  2500  (50^).  The  resulting  pro- 
portion is  .0884  (221/2500)  which  when  rounded  as  a per- 
centage is  equal  to  9.  That  is,  9 per  cent  of  however  many 
tasks  are  to  be  selected  are  taken  from  the  low  frequency- 
moderate  criticality  cell  of  the  matrix.  If  a total  of  20  tasks 
(i. e.,  10  tasks  are  to  be  included  in  the  demonstration;  ac- 
cordingly, a total  of  20  is  to  be  selected  at  this  stage)  is  re- 
quired then  two  tasks  ( 20  x . 0884=  1. 768  rounded  to  2)  would 
be  randomly  selected  from  this  cell.  In  the  case  in  which 
more  tasks  are  designated  to  be  drawn  from  a cell  than  the 
cell  contains  (poor  planning  in  the  original  assignment  of  tasks 
to  levels),  then  the  tasks  should  be  randomly  selected  on  the 
basis  of  relative  proportions,  from  other  cells.  The  sampling 
should  be  done  without  replacement. 


3.4 


Scoring.  For  each  task  selected  as  the  result  of  the 
procedures  of  3,  3,  a recording  or  scoring  sheet  is 
developed.  The  purpose  of  the  scoring  sheet  is  to 
allow  scoring  performance  on  the  individual  subtasks 
of  the  task.  There  are  several  possible  types  of  scor- 
ing sheets.  One  of  the  most  popular  methods  is  a 
sequential  (checklist)  scoring  sheet.  Another  possible 
method  is  through  a technician  recorded  sheet, 

3.4.  1 Sequential  Scoring  Sheet.  To  construct  a sequential 

human  performance  scoring  sheet  for  a task,  the  task 
is  decomposed  into  the  sequential  subtasks  which  must 
be  correctly  performed  if  the  task  is  to  be  completed. 

Each  of  the  subtasks  is  then  sequentially  listed  for 
scoring  by  an  examiner  while  he  observes  the  perform- 
ance of  a technician  on  tne  task.  Time  allowances,  if 
appropriate,  are  also  listed  on  the  recording  sheet. 

An  example  of  such  a recording  sheet  for  a hypothetical 
detection  and  tracking  task  is  presented  as  Figure  2. 

3.4.2  Technician  Recorded  Scoring  Sheet.  The  technician  re- 
corded scoring  sheet  is  useful  when  several  technicians 
can  be  tested  at  once.  The  kinds  of  information  which 
can  be  recorded  by  a technician  include  measurements, 
locations,  or  interpretations.  The  technician  recorded 
scoring  sheet  should  not  test  reading  or  writing  skills -- 
only  actual  performance.  The  written  response  made  by 
the  technician  should  be  limited  to  such  pencil  entries  as 
a checkmark,  underlining,  or  a simple  entry.  Examples 
of  technician  recorded  scoring  sheets  for  two  hypothetical 
tasks,  an  instrument  reading  and  a target  classification 
task,  are  presented  in  Figures  3 and  4. 

3.4.3  Criteria  for  Failure.  For  each  subtask  (or  item  in  a tech- 
nician recorded  scoring  sheet)  the  contractor  should 
develop  fully  objective  criteria  against  which  technician 
performance  may  be  assessed. 


/ 
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3.4 


Scoring.  For  each  task  selected  as  the  result  of  the 
procedures  of  3.  3,  a recording  or  scoring  sheet  is 
developed.  The  purpose  of  the  scoring  sheet  is  to 
allow  scoring  performance  on  the  individual  subtasks 
of  the  task.  There  are  several  possible  types  of  scor- 
ing sheets.  One  of  the  most  popular  methods  is  a 
sequential  (checklist)  scoring  sheet.  Another  possible 
method  is  through  a technician  recorded  sheet. 

3.4.1  Sequential  Scoring  Sheet.  To  construct  a sequential 
human  performance  scoring  sheet  for  a task,  the  task 
is  decomposed  into  the  sequential  subtasks  which  must 
be  correctly  performed  if  the  task  is  to  be  completed. 

Each  of  these  subtasks  is  then  sequentially  listed  for 
scoring  by  an  examiner  while  he  observes  the  perform- 
ance of  a technician  on  the  task.  Time  allowances,  if 
appropriate,  are  also  listed  on  the  recording  sheet. 

An  example  of  such  a recording  sheet  for  a hypothetical 
detection  and  tracking  task  is  presented  as  Figure  2. 

3.4.2  Technician  Recorded  Scoring  Sheet.  The  technician  re- 
corded scoring  sheet  is  useful  when  several  technicians 
can  be  tested  at  once.  The  kinds  of  information  which 
can  be  recorded  by  a technician  include  measurements, 
location,  or  interpretations.  The  technician  recorded 
scoring  sheet  should  not  test  reading  or  writing  skills -- 
only  actual  performance.  The  written  response  made  by 
the  technician  should  be  limited  to  such  pencil  entries  as 
a checkmark,  underlining,  or  a simple  entry.  Examples 
of  technician  recorded  scoring  sheets  for  two  hypothetical 
tasks,  an  instrument  reading  and  a target  classification 
task,  are  presented  in  Figures  3 and  4. 

3.4.  3 Criteria  for  Failure.  For  each  subtask  (or  item  in  a tech- 

nician recorded  scoring  sheet)  the  contractor  shall  develop 
fully  objective  criteria  for  against  which  technician  per- 
formance may  be  assessed. 
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HUMAN  RELIABILITY  EVALUATION  AN/- 
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Figure  2.  Example  of  sequential  scoring  sheet. 
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INSTRUMENT  READING 
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•HU  towwwww 


Figure  3.  Example  of  technician  recorded  scoring  sheet. 


> 

TARGET  CLASSIFICATION 


Technician  Identification!  Date 

Directions!  The  system  is  fully  energized,  checked,  and  working  normally. 

Targets  will  appear,  one  at  a time,  over  the  next  30  minutes. 
As  each  target  appears,  enter  the  time  you  detected  it  and  its 
range,  azimuth,  and  evaluation.  Also,  attempt  to  classify  it. 
Enter  your  answers  in  the  spaces  below.  The  main  thing  is  to 
treat  each  target  as  it  appears.  Work  carefully  but  don't 
waste  time. 


" ■ . i — . r..  -.  r-  .■  : : == 

Time  of 

Target  Detection  Range  Azimuth  Evaluation  classification 


Figure  4.  Additional  example  of  technician  recorded  scoring  sheet. 
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Examples  of  Objective  Criteria 


a.  Completes  circuit  check  within  x minutes 

b.  Torques  to  ^x  pounds 

c.  Detects  target  within  x minutes 


Examples  of  Nonobjective  Criteria 

a.  Expeditiously  completes  circuit  check 

b.  Adjusts  torque  appropriately 

c.  Speedily  detects  target 


3.4,4  Interrater  Reliability.  Task  scoring  which  depends  on 

observation  of  the  technician  as  he  performs  the  task 
or  on  judgments  should  produce  scores  which  are  in 
demonstrable  agreement  when  the  observations  are 
performed  by  different  observers.  Such  between  rater 
(scorer)  data  can  be  collected  during  the  HR  demonstra- 
tion and  may  be  based  on  a 20  per  cent  sample  of  the 
tasks  included  in  the  demonstration.  However,  failure 
to  demonstrate  an  interater  reliability  of  . 85  or  greater 
shall  contitute  failure  of  the  total  demonstration.  The 
interrater  reliability  may  be  quantified  by  correlation, 
analysis  of  variance,  or  other  standard  techniques.  An 
interater  reliability  less  than  . 85  will  require  demonstra- 
tion repetition  and  such  action  as:  further  training  of 
evaluators,  redefinition  of  success  and  failure  points 
within  the  tasks,  selection  of  different  tasks,  and  develop 
ment  of  improved  scoring  sheets. 

An  example  of  the  analysis  of  variance  technique  for 
the  estimation  of  between  rater  agreement  follows; 

Example: 

Assume  that  three  evaluators  have 
observed  the  performance  of  tech- 
nicians performing  four  tasks.  The 
score  given  on  each  task  by  each 
evaluator  is  shown  in  columns  1, 

2,  and  3 below. 
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Task  (N) 


Evaluator  (K) 


1 

2 

3 

TOTAL 

1 

.88 

.85 

.87 

2.60 

2 

.76 

.78 

.77 

2.31 

3 

.94 

.95 

.93 

2.82 

4 

.91 

.90 

.92 

2.73 

TOTAL 

3.49 

3.48 

3.49 

G=  10.46 

S>2  3.06 

3.04 

3.06 

(1)  = 


KN 


109.4116 

3x4 


9.11763 


(2)  = 

(3)  = 

(4) 


r (Ex2 ) = 9.1682 
ET2j  i aq2 


3.49^  + 3.482  + 3.492 


N 

r p2 


= 9.11765 


i 2.602  + 2.312  + 2.822  + 2.732 


9.16713 


SS  between  tasks-  (4)  - (1)  = .04950 
SS  within  tasks  - (2)  - (4)  » .00107 
SS  between  judges-  (3)  - (1)  * .00002 
SS  residual-  (2)  - (3)  - (4)  + (1)-  .00105 

SS  total-  (2)  - (1)  = .05057 


Summary  of  Analysis 


Source  of  Variance 

SS 

df 

MS 

Between  tasks 

.04950 

3 

.01650 

Within  tasks 

.00107 

8 

.00013 

between  judges 

.00002 

2 

- 

residual 

.00105 

6 

- 

TOTAL 

.05057 

11 

Agreement  between  raters- 

^ MS  within 

tasks  _ 

(.00013) 

.01650 

MS  between 

tasks 

I 


r 


» 


t 


» 


t 


t 


t 


3.4.5  Unobtrusive  Measures.  When  the  scoring  depends  on  direct 

rater  observations,  effort  should  be  made  to  assure  that  the 
observations  will  influence  the  ongoing  behavior  minimally. 

4.  Detailed  Requirements:  Conduct  of  the  HR  demonstration. 

4. 1 Task  Selection  and  Order  of  Administration.  A sample  of 

the  tasks  identified  by  the  methods  of  paragraph  3.  3 should 
be  chosen  for  inclusion  in  the  HR  demonstration.  This 
sample  should  be  a 50  per  cent  random  sample  of  the  tasks 
selected  by  methods  of  paragraph  3.  3.  The  eligible  tasks 
should  be  placed  in  a hat.  Then,  the  first  task  to  be  demon- 
strated should  be  selected  and  implemented.  After  the 
demonstration  of  the  first  task  has  been  completed,  the 
second  task  should  be  drawn  and  implemented.  This  pro- 
cedure continues  until  the  sampling  requirements  have 
been  met;  i.  e.  one  half  of  the  tasks  have  been  selected  and 
demonstrated. 

In  some  cases  the  Government  may  specify  a special 
sample  of  tasks  (e.  g. , all  tasks  of  a given  level  of 
criticality)  for  inclusion  in  the  final  HR  demonstration. 

In  these  cases,  the  sample  of  tasks  should  include  all 
such  tasks  and  a representative  (random)  sample  of  the 
remaining  tasks. 

4.  2 Test  Methods. 

4.2.  1 Task  Presentation.  The  tasks  selected  for  inclusion  in  the 

HR  demonstration  should  be  presented  in  the  random  order 
in  which  they  are  drawn.  The  tasks  which  the  technicians 
will  be  asked  to  perform  during  the  HR  demonstration 
should  be  presented  in  a context  that  duplicates  to  the  extent 
possible  that  which  will  be  found  during  likely  applications  of 
the  equipment/subsystem/system. 

4.2.2  Non-Random  Sequences.  In  the  unlikely  event  that  equip- 
ment set  up  requirements  may  dictate  schedule  rearrange- 
ment, the  random  sequence  of  tasks  may  be  rearranged. 

4.2.3  Inter-Task  Presentation  Time.  Whenever  possible,  the 
tasks  should  be  presented  by  the  state  of  the  equipment/sub- 
system/system  and  not  verbally  by  an  administrator.  The 
time  between  task  presentations  will  be  a function  of  the 
time  required  for  equipment  set  up.  Long  breaks  between 
tasks  should  be  avoided  and  may  constitute  a basis  for  a 
non-random  task  presentation  sequence. 
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Equipment  Failure  during  Demonstration.  In  the  event 
of  an  equipment  failure  during  a task  demonstration,  the 
demonstration  should  be  interrupted  and  after  restora- 
tion of  the  equipment  to  operational  status,  a substitute 
task  should  be  drawn  from  the  bank  of  tasks.  The  sub- 
stitute task  will  serve  as  a replacement  for  the  original 
task. 

Operator  Teams.  In  the  event  that  more  than  one  operator 
or  one  maintainer  team  is  employed  for  the  HR  demonstra- 
tion, the  interaction  among  the  individuals  and  the  teams 
should  be  controlled.  The  various  teams  should  function 
independently.  No  one  individual  should  be  a member  of 
more  than  one  team. 

Intrateam  Relationships.  The  members  of  each  team  should 
function  with  a level  of  autonomy  similar  to  that  expected 
during  actual  operation  of  the  equipment/subsystem/system. 
Each  team  member  should  assume  a level  of  responsibility 
as  might  be  expected  in  actual  applications. 

Task  Presentation  Directions.  Task  instructions  should  be 
presented  in  a standard  form  and  should  be  read  aloud  by  an 
examiner  to  the  participating  technician(s)  while  the  techni- 
cian(s)  read(s)  the  directions  silently.  These  directions 
should  contain  all  information  required  by  the  technician  for 
completing  the  task  and  no  additional  information.  An 
example  of  such  directions  is  presented  in  Figure  5. 

Information  Provided  to  Technicians.  Once  started  on  a task, 
the  technician  performing  the  task  should  receive  no  help  or 
guidance  from  the  evaluators,  test  administrators  or  other 
personnel  present.  Administrators  will  provide  only  that  in- 
formation which  is  included  in  the  task  presentation  direc- 
tions. 'Where  inadequate  performance  (subtask  failure)  is 
not  noted  by  the  technician,  it  should  be  pointed  out  to  him 
only  if  subsequent  subtask  performance  will  be  affected. 


TASK  PRESENTATION  DIRECTIONS 
TARGET  DETECTION  AND  TRACK 


Purpose:  The  purpose  of  this  aspect  of  the  HR  demonstration  is  to 

demonstrate  the  ability  of  operators  to  detect  and  track 
targets  on  the  AN/ system. 

Your  Task:  When  I say  "start,"  seat  yourself  at  the  equipment  and 

monitor  the  system  in  a "normal"  manner.  Report,  as  early 
as  you  are  sufficiently  confident,  the  range  and  azimuth 
of  any  target  you  detect  and  prosecute  each  target  includ- 
ing full  lock  on  and  track  over  the  course  of  the  demonstra- 
tion. Be  sure  to  folic*  all  required  procedures  for  assur- 
ing early  detection  and  full  track.  If  you  lose  a target, 
go  through  the  normal  reacquisition  procedures.  If  you  have 
any  questions  ask  them  now.  No  questions  will  be  answered 
during  the  demonstration. 

Equipment:  The  AN/ has  been  turned  on  and  is  operational. 

However,  you  may  make  any  adjustments  you  desire. 

Scoring:  You  will  be  scored  on  how  quickly  you  report  each  target, 

how  accurately  you  report  it,  and  how  well  you  prosecute  it. 

You  will  be  downgraded  if  you  fail  to  follow  correct  procedures 
or  if  you  violate  equipment  or  human  safety  precautions. 


Do  you  have  any  questions?  Start. 


Figure  5.  Example  of  task  presentation  directions. 


Conditions  of  Test.  The  testing  conditions  should 
realistically  duplicate  the  various  conditions  under 
which  the  system  operates.  The  resources  made 
available  for  use  should  be  those  supplies  and  equip- 
ment which  are  expected  to  be  commonly  available 
when  the  system  is  in  actual  use. 


4.  3.  1 Specification  of  Resources.  The  resources  that  will  be 

made  available  during  the  final  HR  demonstration  should 
be  listed  prior  to  the  onset  of  the  demonstration.  Procuring 
agency  approval  of  the  list  should  be  obtained. 

4.  3.  1.  1 Apparatus.  Testing  will  be  done  on  the  actual  equipment/ 
subsystem/system  under  consideration.  This  equipment 
should  be  identical  in  all  major  respects  to  the  actual 
equipment  planned  for  delivery.  Major  respects  are  those 
aspects  which  might  affect  human  performance. 

4.  3.  1.2  Test  Equipment,  Tools,  and  Supplies.  Only  standard  test 
equipment,  tools,  and  supplies  which  are  expected  to  be 
available  during  the  normal  operation  of  the  equipment/ 
subsystem/system  should  be  permitted  for  use  during  the 
HR  demonstration. 

4.  3.2  Time  and  Duration  of  Test.  The  time  of  day  of  testing  and 

the  duration  of  testing  on  any  one  day  need  not  be  consonant 
with  the  time  of  day  and  length  of  operator/ team  shift 
anticipated  for  the  equipment/  subsystem/system  in  actual 
application.  However,  no  one  operator/ team  should  be 
tested  for  a length  of  time  greater  than  that  anticipated  for 
actual  application.  Similarly,  turnover  of  personnel 
within  the  length  of  time  of  a typical  work/ shift  should  be 
avoided. 

4.  3.  3 Technician  and  Evaluator  Selection  and  Training.  The 

validity  of  the  HR  demonstration  is  dependent  on  the  qualifica- 
tions and  the  degree  of  training  of  both  the  technicians  and 
the  evaluators.  All  aspects  of  such  selection  and  training 
should  be  carefully  documented  and  detailed  and  be  subject 
to  Governmental  approval. 

4.  3.  3.  I Technician  Qualifications.  Personnel  equivalent  to  and 

representative  of  those  who  are  anticipated  to  be  operators 
on  the  equipment/system  should  be  used  in  the  HR  demonstra- 
tion. Equivalency  should  be  assessed  in  terms  of  such 
variables  as:  experience,  aptitude,  and  schooling.  Physical 
characteristics,  such  as  size  and  weight,  should  be  con- 
sidered whenever  relevant. 
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Where  possible  standardized  tests  should  be  used  to 
assure  similarity  between  personnel  in  the  test  situation  and 
in  the  operational  military  setting.  The  Armed  Service 
Vocational  Aptitude  Battery  (ASVAB)  requirements  and 
norms  for  personnel  in  comparable  occupations  in  the 
military  should  constitute  the  basis  for  assessing  the  simi- 
larity of  demonstration  technicians  to  military  personnel 
available  for  assignment  to  the  equipment/subsystem/ system. 
Both  upper  and  lower  limits  should  be  considered. 

In  this  regard,  the  contractor  should  submit  for  approval 
the  ASVAB  scores  and  the  other  factors  to  be  considered  in 
assessing  equivalency,  along  with  the  criteria  to  be  employed 
in  establishing  equivalency. 

4.  3.  3.  2 Technician  Training.  The  system  specific  training  given 
to  the  technicians  who  participate  in  the  HR  demonstration 
should  not  be  greater  in  content  or  extent  than  that  anticip- 
ated for  training  personnel  who  will  man  the  system  in 
actual  operation. 

Any  system  specific  training  received  by  the  technicians 
who  participate  in  the  HR  demonstration  should  be  fully 
documented  in  terms  of  the  specific  content  of  the  course  and 
the  time  allotted  for  each  topic.  The  criteria  for  evidence  of  • 
mastery  of  the  training  program  should  also  be  specified. 

This  information  should  be  submitted  at  least  30  days  prior 
to  the  demonstration  and  include  training  anticipated  during 
the  period  between  the  submission  and  the  HR  demonstration. 

4.  3.  3.  3 Evaluator  Selection.  The  HR  demonstration  administrators/ 
evaluators  should  be  supplied  by  the  contractor  and  should 
have  professional  training  in  testing  and  relevant  professional 
training  or  professionally  supervised  experience  in  admin- 
istering performance  tests.  A background  in  industrial 
psychology  is  particularly  germane  to  this  requirement. 
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4.  3.  3.  4 Evaluator  Training.  Each  evaluator/ administrator  should 
be  fully  trained  by  the  contractor  in  the  areas  of  the  testing 
in  which  he  will  be  involved.  This  training  should  include 
an  overview  of  the  purpose  and  procedures  of  the  demonstra- 
tion, requirements  for  assuring  standardization  of  conditions 
of  testing  and  scoring,  and  typical  errors.  The  evaluators 
should  be  appropriately  trained  in  the  operation  of  the  equip- 
ment/subsystem/system under  consideration.  These  train- 
ing procedures  should  be  fully  documented  and  submitted 
along  with  the  documentation  of  the  technician  training. 


4.3.4  Control  of  Testing  Conditions.  All  aspects  of  the  testing 

conditions  should  be  fully  standardized.  This  includes, 
but  is  not  limited  to  the:  timing  of  the  various  sections  of 
the  tests,  instructions  given  to  the  technicians,  methods  of 
dealing  with  questions  that  may  arise,  recording  of  anoma- 
lies, and  the  nature  and  method  of  collecting  data.  The 
objective  aspects  should  be  specified  in  writing.  The  less 
objective  aspects  should  be  described  in  writing  and  further 
described  in  the  administrator/evaluator  training  period. 

4.  3.  4.  1 Demonstration  Manual.  To  aid  in  achieving  the  goals  of 

standardization  and  objectivity,  a manual  should  be  developed 
which  contains  all  the  demonstration  conduct  information 
necessary  to  assure  these  goals.  The  manual  should  contain 
all  information  necessary  for  orderly  conduct  of  the  demonstra- 
tion including,  but  not  limited  to,  unambiguous  statements  of 
the  rules  for  conduct  and  standardization  of  the  HR  demonstra- 
tion, the  roles  and  specific  responsibilities  of  the  various 
participants,  the  time  limits  for  each  task,  the  rules  and 
controls  which  will  be  observed,  and  the  overall  schedule. 

In  case  of  conflict  between  the  demonstration  manual  and 
these  Guidelines,  the  Guidelines  should  pertain.  The 
manual  should  be  submitted  to  the  Government  for  approval 
at  least  45  days  prior  to  the  start  of  the  HR  demonstration. 

4.  3.4.  2 Chief  Test  Administrator.  The  Government  should  appoint 
a Chief  Administrator  who  represents  the  Government  and 
who  maintains  administrative  and  control  responsibility 
over  the  conduct  of  the  HR  demonstration.  Decisions  rela- 
tive to  the  problems  which  arise  during  the  conduct  of  the 
demonstration  should  be  the  responsibility  of  the  Chief  Ad- 
ministrator and  be  final. 

3 
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4.  3.4.  3 Arrangement  of  Demonstration  Area.  The  demonstration 
should  be  conducted  in  a private  area  which  allows  unin- 
terrupted conduct  of  the  required  tests.  While  it  is  recog- 
nized that  contractor  and  Governmental  observers,  repre- 
senting various  positions  of  responsibility  in  the  development 
of  the  equipment/ subsystem/ system  under  development,  will 
want  to  observe  the  demonstration,  the  comingling  of  persons 
actually  involved  in  the  demonstration  with  uninvolved  observ- 
ers should  not  be  permitted.  Similarly,  the  comingling  of 
Governmental  and  contractor  observers  is  discouraged.  A 
suggested  arrangement  of  the  demonstration  area  is  presented 
in  Figure  6, 
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FIGURE  6.  SUGGESTED  ARRANGEMENT  OF  DEMONSTRATION  AREA 


4.  3. 4.  4 


Timing.  The  total  time  for  the  completion  of  each  demonstra- 
tion task  should  be  kept  by  an  individual  who  is  acceptable  to 
both  the  Government  and  the  contractor.  In  the  event  that 
the  time  for  total  task  completion,  as  measured  by  this  in- 
dividual, does  not  agree  with  the  time  measured  by  the 
scorer(s),  the  Chief  Administrator  should  reconcile  the 
difference(s). 

4.  3.4.5  Engineering  Support.  Engineering  support,  test  equipment, 
replacement  parts,  and  the  like,  necessary  for  maintaining 
the  equipment/ subsystem/ system  under  test  in  a proper 
operating  condition  during  the  HR  demonstration,  should  be 
provided  by  the  contractor. 

4.4  Variations.  It  may  not  be  possible  to  comply  in  every  respect 
with  the  requirements  detailed  above.  For  example:  (a)  it 
may  be  extremely  hazardous  to  have  a technician  perform 
certain  tasks,  (b)  it  may  be  impossible  to  simulate  realistic- 
ally the  real  world  conditions  under  which  one  or  several 
tasks  will  be  typically  conducted,  or  (c)  training  technicians 
(as  described  in  section  4.  3.  3.  2)  may  be  prohibitive  in  terms 
of  time  or  personnel  available.  In  such  cases,  the  contractor 
should  propose  a substitute  (task,  simulation,  technician 
type,  training,  etc. ) to  attempt  to  meet  the  demonstration 
requirements.  These  proposals  should  describe  and  detail 
the  reason  for  the  substitute  arrangements  and  the  anticipated 
impact  of  these  substitutions  on  the  HR  of  the  equipment/ sub- 
system/ system  under  test.  The  proposals  should  be  imple- 
mented in  the  course  of  the  final  HR  demonstration,  if  approved 
by  the  Government. 

4.5  Control  of  Test  Materials.  The  purpose  of  the  HR  demonstra- 
tion becomes  meaningless  if  the  test  materials  are  compro- 
mised in  any  way.  The  contractor  should  implement  a strict 
control  program  to  assure  that  the  materials  are  compromised 
in  no  way  and  that  the  demonstration  technicians  have  no  prior 
specific  information  relative  to  demonstration  and  test  content. 
Evidence  of  lack  of  control  over  test  materials  should  consti- 
tute a basis  for  automatic  failure  of  the  demonstration. 
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Audit  Trial.  Throughout  the  development  and  implementation 
of  the  HR  demonstration,  considerable  responsibility  for 
integrity  assurance  rests  with  the  contractor.  However,  a 
full  audit  trial  should  be  left  by  the  contractor  so  that  the 
steps  leading  to  the  various  end  products  can  be  fully  evaluated 
by  the  Government.  Such  audit,  at  the  option  of  the  Govern- 
ment, may  take  place  periodically  over  the  various  steps  in 
HR  demonstration  development  and  may  include  actual  inter- 
views with  participating  personnel  as  well  as  review  of  support- 
ing data  and  documentation. 

Analysis  and  Reporting  of  Demonstration  Data.  The  contractor 
should  be  responsible  for  analyzing  and  documenting  the 
results  of  the  HR  demonstration.  No  extensive  written  report, 
in  the  sense  of  the  usual  "technical  report,  " is  necessary  or 
desirable.  Rather,  a set  of  forms  should  be  submitted  which 
will  contain  the  data  accumulated  during  each  step  of  the  process. 
Examples  of  the  required  forms  and  their  use  are  given  below. 
Substitute  forms  containing  essentially  the  same  information 
as  that  shown,  but  differing  in  format,  should  be  acceptable. 

Time  for  Form  Completion  and  Submission.  Each  form  should 
be  submitted  immediately  after  its  completion. 

Forms.  The  categories  of  forms  to  be  completed  are: 

Task  identification  and  description  (Form  HR-1)-- 
submitted  during  preparation  phases. 

Selection  of  demonstration  tasks  (Form  HR-2)-- 
submitted  during  preparation  phases. 

Human  reliability  demonstration  data  collection  forms 
(Form  HR-3)--submitted  during  preparation  phase. 

Human  reliability,  human  availability,  and  human 
MTTR  (Form  HR-4)--submitted  after  demonstration. 

Technician  qualifications  (Form  HR-5)--submitted 
after  demonstration. 

Interrater  reliability  (Form  HR-6)--submitted  after 
demonstration. 


5.2.1  Form  HR- 1.  Form  HR -1,  or  an  equivalent,  should  be 
used  for  Identification  of  the  operator/ maintainer  involved 
tasks  In  the  system,  the  frequency  of  occurrence  of  each  task 
and  the  criticality  of  each  task.  All  data  required  by  this 
form  should  be  provided. 

5.2.2  Analyst(s).  The  name  or  names  of  the  individual(s) 
completing  the  form(s)  should  be  indicated. 

5.2.3  Source  of  Data.  The  reference  materials,  specifications, 
reports,  Individuals,  and  so  on  used  as  information  sources 
should  be  indicated. 

5.2.4  Task  Number.  A unique  identification  number  should  be 
assigned  to  each  task  listed. 

5.2.5  Name  or  Description.  A prose  description  sufficiently  detailed 
to  distinguish  between  similar  tasks  should  be  entered  for 
each  task. 

5.2.6  Equlpment(s)  Involved.  The  equipment  units  required  for 
demonstrating  performance  of  this  task  should  be  identified. 

5. 2. 7 Frequency.  The  relative  frequency  of  occurrence  of  each 
task  should  be  identified.  Frequency  should  be  scaled  in  the 
same  manner  as  criticality. 

5. 2.  8 Criticality.  The  criticality  of  each  task  should  be  scaled  as 

low  (L),  moderate  (M),  or  high  (H). 

5. 2. 0 Duration.  The  nominal  or  expected  duration  of  each  task 
should  be  indicated  in  hours  and  minutes. 

5. 3 Selection  of  Human  Reliability  Demonstration  Tasks. 

5. 3. 1 Form  HR -2.  Form  HR -2,  or  an  equivalent,  should  be  used 
to  document  the  selection  of  the  tasks  to  be  sampled.  Note 
that  the  goal  here  is  to  identify  twice  the  number  of  tasks 
that  will  be  Included  in  the  final  HR  demonstration. 
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HUMAN  RELIABILITY  DEMONSTRATION  FORM  HR-1,  TASK  IDENTIFIC 


Figure  7.  Example  of  Form  HR-1,  Task  identification  and  Description 


5.3.2 


5.  3.  3 


5.3.4 


5.3.3 


Task  Sorting.  All  task  numbers  entered  on  the  Form  HR-2 
should  be  sorted  into  the  appropriate  frequency-criticality 
cell  as  demonstrated  in  the  example  of  Section  II, Form  HR-2. 

Task  Total.  The  total  number  of  tasks  which  fall  in  each 
level  of  criticality  and  each  level  of  frequency  should  be 
computed  and  entered  in  the  appropriate  location  in  Section 
II  of  Form  HR -2.  The  sum  of  the  totals  for  each  level  of 
criticality  must  agree  with  the  sum  of  the  totals  for  each  level 
of  frequency  and  be  equal  to  the  total  number  of  tasks. 

Cross  Product  Total.  The  criticality  and  frequency  level 
totals  should  be  entered  in  the  appropriate  locations  in 
Section  III  of  Form  HR-2.  The  appropriate  criticality  level 
total  should  be  multipled  by  the  appropriate  frequency  level 
total  to  produce  a frequency-criticality  cross  product  for 
each  cell.  In  the  example,  the  low  criticality  level  total  of 
16  and  the  low  frequency  level  total  of  14,  yields  a resultant 
cross  product  equal  to  224  (16  x 14*  224).  This  frequency- 
criticality  cross  product  should  be  computed  for  each  cell 
and  the  resultant  value  entered  in  the  cell.  The  sum  of  the 
nine  resulting  cross  products  should  be  entered  in  the  loca- 
tion labelled  "Cross  Product  Total. " 

Desired  Task  Number  Entry.  The  desired  number  of  tasks 
to  be  selected  should  be  entered  in  the  location  labelled 
"Number  of  Tasks  to  be  Selected.  " Note  that  this  number 
must  be  at  least  two  times  the  number  to  be  demonstrated. 

The  actual  tasks  to  be  demonstrated  will  be  50  per  cent  random 
sample  (without  replacement  of  the  selected  tasks).  The 
sample  is  drawn  at  the  time  of  the  demonstration. 


« 
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AN/ SYSTEM 

t 

HUMAN  RELIABILITY  DEMONSTRATION  FORM  HR-2,  TASK  SELECTION  FORM 

Date Analyst (s) 



f 

SECTION  I:  LIST  OF  TASKS  SELECTED 

TASK  NUMBER  DESCRIPTION 

* 


* 


I 


Figure  8.  Example  of  Form  HR-2,  Task  Selection. 
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SECTION  II:  FREQUENCY-CRITICALITY  MATRIX 


FREQUENCY 


CRITICALITY 

LOW 

MODERATE 

HIGH 

LOW 

*1 

1,19,37,38,39 

6,20,21,40,41 

42 

4,22,28,29,34 

MODERATE 

10,14,15,16,17 

18 

2,8,36,43 

3,5,27,35,44, 

45,46,47,48,49 

50 

HIGH 

11,12,13 

7,9,30,31,32 

33 

23,24,25,26 

TOTALS 

14 

16 

20 

TOTALS 


21 


13 


*1  Actual  task  numbers  which  fall  in  cell 

*2  Total  number  of  tasks  in  the  respective  row  or  column 

SECTION  III:  FREQUENCY -CRITICALITY  CROSS  PRODUCTS 


FREQUENCY 


CRITICALITY 

LOW 

MODERATE 

HIGH 

TOTALS 

LOW 

224*2  3*4 

256  3 

320  4 

*1 

16 

MODERATE 

294  4 

336  4 

420  5 

21 

HIGH 

182  2 

208  2 

260  3 

13 

TOTALS 

14 

16 

20  TOTAL  50 

CROSS  PRODUCT  TOTAL  (CPT)  2 500* 3 


NUMBER  OF  TASKS  TO  BE  SELECTED 
TTBS 


CELL  WEIGHT" 


CPT 


30 

2500 


(TTBS)  = 30 


= .012 


*1  - Row  and  column  totals  are  taken  from  Section  II 
*2  - Cross  product  of  row  total  and  column  total 
*3  - Sum  of  cross  products  across  all  cells 

*4  - Rounded  product  (to  nearest  integer)  of  cell  weight  and  cell 
frequency-criticality  cross  product. 


Figure  8.  (con't. ) 
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5.  3.6  Cell  Weight.  The  frequency-criticality  cell  weight  should 

be  computed  as  the  number  of  tasks  to  be  selected  divided 
by  the  cross  product  total.  The  cell  weight  should  be 
multiplied  by  the  frequency-criticality  cross  product  of 
each  cell  to  determine  the  number  of  tasks  to  be  selected 
from  each  cell.  The  resulting  product  should  be  rounded  to 
the  nearest  integer.  If  more  tasks  are  to  be  selected  from  a 
cell  than  are  available  for  selection,  then  the  tasks  should 
be  randomly  selected,  on  the  basis  of  relative  proportions, 
from  other  cells. 

5.  3.7  Task  Bank.  The  sample,  as  derived  in  5.  3,  determines  the 

task  bank  from  which  tasks  are  drawn  for  the  HR  demonstra- 
tion. The  names  of  the  tasks  in  the  task  bank  should  be 
entered  in  Section  I of  Form  HR -2  by  number  and  description. 

5.  4 Form  HR-3.  Form  HR-3  should  be  used  to  document  the 

collection  of  human  reliability  demonstration  data.  There 
should  be  one  such  form  developed  for  each  task  in  the  total 
sample  to  be  available  for  the  final  demonstration  (i.  e., 
one  form  for  each  task  listed  in  Section  I of  the  HR-2  form). 
The  nature  of  these  forms  is  not  prescribed.  However,  it  is 
anticipated  that  the  forms  will  be  substantially  similar  to 
those  presented  in  Figures  2,  3,  and  4 of  these  Guidelines. 
Prior  approval  of  the  Government  should  be  required  for  gross 
deviations  from  these  forms.  Regardless  of  the  format 
employed,  each  form  HR -3  should  contain  the  information 
described  below. 

5.4.1  Observer.  The  name  and  any  other  relevant  identification 

required  to  identify  the  observer  who  completed  the  form 
should  be  indicated.  Only  one  person  should  be  identified. 

If  multiple  observers  are  used,  they  should  independently 
complete  different  forms. 

5.  4.2  Task  Number.  The  task  should  be  identified  by  a number 

which  is  the  same  as  that  listed  for  the  task  on  Form  HR-2. 

5.4.3  Name  of  Task.  The  task  should  be  identified  by  the  same 

name  as  that  listed  on  Form  HR-1. 
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5.4.4  Technician  Identification.  Relevent  identification  of  each 
technician  involved  in  the  task  demonstration  should  be 
provided. 

5.4.5  Time.  Start  and  stop  times  for  the  task  and  subtasks 
(where  appropriate)  should  be  expressed  in  minutes  and 
seconds. 

5.4.6  Success  or  Failure.  The  outcome  of  each  subtask  should  be 
criterion  referenced  and  should  be  indicated  as  S or  F 
(success  or  failure)  or  an  equivalent. 

5.4.7  Fix-Up.  Subtask  repetition  on  fix-up,  necessitated  by  in- 
adequate performance  on  the  first  attempt  is  termed  fix-up 
performance.  Start  time,  stop  time,  and  the  success  or 
failure  of  any  fix-up  attempts  should  be  recorded.  In  the 
case  of  a failure  on  the  fix-up,  a second  fix-up  attempt 
should  not  be  allowed. 

5.4.  8 Comments.  Any  unusual  or  potentially  relevant  observations, 

especially  those  which  lend  insight  into  the  accuracy  of  the 
scoring  and  the  validity  of  the  observations,  should  be  recorded. 
The  reference  point  to  the  relevant  sequenced  subtask  should 
be  indicated  by  an  asterisk  (*)  followed  by  a number  (e.  g. , 

*1,  *2,  *3)  if  more  than  one  comment  is  recorded. 

5.5  Form  HR-4.  Form  HR -4  or  an  equivalent  should  be  used  for 

calculating  and  reporting  human  reliability,  human  availa- 
bility, and  human  mean  time  to  repair. 

5.5.  1 Date.  The  date  on  which  the  form  was  completed  should  be 

entered. 

5,5.2  Analyst.  The  name(s)  of  the  individual(s)  performing  the 

data  summarization  and  those  who  calculated  the  human 
reliability,  human  availability,  and  human  MTTR  values 
should  be  entered. 

5.5.  3 Number  of  Subtask  Failures.  The  total  number  of  failures 

of  all  subtasks  demonstrated  should  be  indicated.  This 
number  is  determined  by  summing  the  number  of  subtask 
failures  indicated  on  each  HR-3  Form. 
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5.5.4 


5.5.5 


Number  of  Attempts.  The  number  of  attempts  at  performing 
all  subtasks  should  be  indicated.  This  number  is  the  sum  of 
both  successes  and  failures,  including  fix-ups,  across  all 
HR-3  forms. 


Human  Reliability  (HR).  The  human  reliability  yielded  by  the 
demonstration  should  be  completed.  For  example,  if  the 
number  of  failures  is  equal  to  41  and  the  number  of  attempts 
is  equal  to  503,  then  HR  is  equal  to  . 92. 


HR=  1 - 


Number  of  Subtask 

Failures 41 

Number  of  Attempts  ” 503 


5.  5.  6 


5.5.7 


5.5.8 


5.5.9 


Unmanned  Station  Hours.  The  number  of  unmanned  station 
hours  should  be  given.  This  value  is  the  total  time  by  which 
the  demonstrated  tasks  extended  beyond  the  nominal  duration 
(and  thereby  could  have  caused  delays  in  the  performance  of 
other  jobs). 

Total  Mission  Man  Hours.  The  total  mission  man  hours  should 
be  entered.  This  value  is  the  total  man  hours  taken  to  complete 
all  tasks. 


Human  Availability  (HAVAIL).  The  calculated  human  avail- 
ability should  be  entered.  If  unmanned  station  hours  is  equal 
to  10  minutes  and  total  mission  man  hours  is  equal  to  240 
minutes,  then  human  availability  is  equal  to  . 96. 


HAVAIL  >= 


Unmanned  Station 

Hours 

Total  Mission 
Man  Hours 


10 

240 


= .96 


Fix-Up  Time.  The  total  time  spent  in  fix-up  should  be  computed 
and  entered. 


5.  5. 10  Number  of  First  Attempt  Failures.  The  number  of  subtasks 
failed  on  the  first  attempt  should  be  entered. 

5.5. 11  Human  Mean  Time  to  Repair  (HMTTR).  The  human  mean  time 
to  repair  should  be  entered.  This  value  is  the  amount  of  time 
spent,  on  the  average,  to  fix-up  tasks  performed  inadequately 
or  unsuccessfully  on  the  first  attempt.  If  the  total  fix-up  time 
is  equal  to  160  minutes  and  the  number  of  failures  is  equal  to 
42,  then  HMTTR  is  equal  to  3.  81  minutes. 
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AN/  SYSTEM 

HUMAN  RELIABILITY  DEMONSTRATION  FORM  HR-4,  HUMAN  RELIABILITY, 
HUMAN  AVAILABILITY,  AND  HUMAN  MTTR 


Date 


Analyst (s) 


Section  I:  Summary  of  Results 


1.  Obtained  Human  Reliability 

2.  Obtained  Human  Availability 

3.  Obtained  Human  MTTR 


Section  II:  Detailed  Calculations 


HUMAN  RELIABILITY  (HR) 

a*  Number  of  subtask  failures* 
b=  Number  of  subtask  attempts* 


I 1 

I .1 


HR*1-  Hr)  -[ 


HUMAN  AVAILABILITY  r 

a=  USH*  Unmanned  station  hours*  l 1 

b=  TMMH*  Total  mission  man  hours*  [ ] 


HAVA  I L*  1 - |-g-J  = f I 


HUMAN  MEAN  TIME  TO  REPAIR  (HMTTR). 

a*  Total  time  spent  in  fix-up*  ! i 

b=  Number  of  tasks  failed  on  first  trial 


HMTTR* 


[ 


Figure  9.  Example  of  Form  HR-4,  Human  Reliability,  Human  Availability,  and 
Human  MTTR. 
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Total  Fix-Up 

HMTTR-  „ T‘me  . ■ '6.°  = 3.81 

Number  of  42 

Failures 

» 

5.  6 

Form  HR -5.  Information  concerning  the  training  and 
qualifications  of  the  technicians  participating  in  the 
demonstration  should  be  recorded  on  Form  HR -5,  or 
an  equivalent.  There  should  be  one  Form  HR -5  for 
each  technician  involved. 

t 

5.6.  1 

Date.  The  date  of  form  completion  shoula  be  entered 
on  the  form. 

5.6.2 

Page.  Each  Form  HR -5  should  be  sequentially  numbered 
and  the  total  number  of  forms  should  be  entered. 

t 

5.6.  3 

Analyst.  The  individual  completing  the  form  should  be 
identified  by  name. 

5.6.4 

Source  of  Information.  The  records,  forms,  personnel  or 
other  sources  consulted  for  obtaining  the  technician  quali- 
fication information  should  be  indicated. 

X 

5.  6.5 

Technician's  Name.  The  name  of  the  technician  for  whom 
the  form  pertains  should  be  entered  along  with  a unique 
code  number. 

1 ;; 

5.  6.6 

Highest  Educational  Achievement.  The  highest  grade  level 
completed  by  the  named  technician  should  be  indicated  in  the 
appropriate  entry  location.  Equivalency  attainments  should 
be  considered  along  with  formal  education. 

5.6.7 

Training  and  Experience  Specific  to  System.  The  specific 
training  and  experience  of  the  named  technician  in  the 
theory,  operation,  and  maintenance  of  the  equipment/ sub- 
system/system  involved  should  be  fully  described. 

5.  6.  8 

Related  Training  and  Experience.  Any  training  and  experience 
having  implications  to  the  tests  on  hand  should  be  given  here. 

5.  6.  9 

Technical  and/or  Vocational  Training.  All  technical,  voca- 
tional, and  on  the  job  training  (including  training  on  the 
current  equipment/subsystem/system)  completed  by  the 
named  technician  should  be  entered  on  the  form  HR-5. 
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HUMAN  RELIABILITY  DEMONSTRATION  FORM  HR-5,  TECHNICIAN  QUALIFICATIONS 

Analyst  ( s ) Date 

Technician's  Name  Code  No. 

Source (s)  of  Information 


Highest  Educational  Achievement  (grade  level)  

Technical  and/or  Vocational  Training  (date,  schools,  and  course  names) 
DATE  SCHOOL  COURSE  NAME 


Training  and  Experience  Specific  to  System 


Related  Training  and  Experience 


Raw  Scores,  Percentile  Equivalents,  and  Norms  on  Standardised  Testa 
TEST  RAW  SCORE  PERCENTILE  EQUIVALENT 


COMMENTS i 


Figure  10.  Example  of  Form  HR-5,  Technician  Qualifications. 
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5.6,  10 

Raw  Scores,  Percentile  Equivalents,  and  Norms.  Raw 
scores  and  percentile  equivalents  on  standardized  tests 
for  the  technician  under  consideration  should  be  given 
along  with  a statement  of  the  norms  used  to  derive  the  per- 
centile equivalents. 

& 

5.6.  11 

Comments.  Any  additional  information  relevant  to  the 
qualifications  of  the  named  technician,  or  his  similarity 
to  real  system  operators  should  be  indicated.  Additional 
pages  may  be  added  as  required. 

5.7 

Form  HR-6.  Form  HR -6,  or  an  equivalent,  should  be 
employed  for  recording  the  data  and  computations  used  in 
determining  interrater  reliability. 

5.7.1 

Analyst,  The  name  of  the  individual  collecting  the  data 
and  performing  the  calculations  for  the  interrater  relia- 
bility determination  should  be  entered. 

A » 

5.7.2 

Task  Number.  The  task  identifying  numbers,  as  indicated 
on  Form  HR-1  should  be  indicated  in  the  appropriate  loca- 
tion. 

5.7.3 

Evaluator.  The  score  produced  by  each  evaluator  on  each 
task  considered  within  the  interrater  reliability  evaluation 
should  be  entered. 

5.7.4 

Computations.  The  procedure  used  to  calculate  interrater 
reliability  should  be  completely  shown.  Intermediate  steps 
sufficient  to  verify  the  computations  should  be  included. 

5.7.5 

Reliability.  The  computed  interrater  reliability  should  be 
entered.  Any  data  available  concerning  the  statistical 
significance  of  the  reliability  should  be  indicated. 

5.7.6 

Comments.  Any  remarks  concerning  the  reliability  data 
collection  and  calculation  should  be  entered.  Specific 
reasons  for  any  severe  discrepancies  between  judges  should 
be  entered,  whenever  known. 
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’ORM  HR-6,  INTERRATER  RELIABILITY 

Dat  e 

EVALUATOR 
"5 4 


5 


Statistical  Considerations.  Since  the  total  task  population 
for  an  equipment/ subsystem/ system  can  not  be  tested 
in  the  HR  demonstration,  a statistical  inference  must  be 
made  concerning  the  attainment  of  the  desired  level  of 
HR  based  on  the  sample  of  the  events  demonstrated.  The 
goal  of  the  statistical  analysis  of  the  HR  demonstration 
data  is  to  verify  that  the  HR  is  equal  to  some  specified 
level  (within  some  tolerance  and  with  a given  level  of 
confidence). 

Any  acceptable  and  standard  statistical  technique  may 
be  employed  to  establish  within  a 10  per  cent  confidence 
limit  the  error  bounds  around  the  human  reliability  attained 
in  the  HR  demonstration.  One  method  for  accomplishing  this 
is  based  on  the  standard  error  of  the  proportion  of  subtasks 
accomplished  successfully  during  the  demonstration  and  on 
the  acceptable  confidence  interval. 

If  . 895  is  the  required  HR  and  an  HR  of  . 92  has 
been  demonstrated,  has  the  . 895  requirement 
been  met?  The  calculation  is  based  on  the  stand- 
ard error  of  the  proportion  and  on  the  confidence 
range  which  is  acceptable.  The  standard  devia- 
tion of  a proportion , a , is  equal  to  ^ pq  where 
p is  the  proportion  of  subtasks  performed  success - 
fully  and  q is  equal  to  1 - p.  In  this  case,  o - 
^ . 92  x . 08  = .271.  The  standard  error  of 
proportion  is  equal  to: 


/ 


If  we  assume  an  N (i.  e.,  number  of  subtasks) 
of  100,  then  <fp  is  equal  to  .0271.  Reference  to 
a z table,  (that  is,  a table  of  the  normal  dis- 
tribution) indicates  that  2,97  times  the  standard 
error  will  include  99  per  cent  of  the  population. 

Since  2.79  times  .0271  is  .076,  we  can  say  that 
we  are  99  per  cent  confident  that  the  real  over- 
all human  reliability  is  between  . 844  (lower  bound) 
and  . 996.  Accordingly,  the  criterion  of  . 895  HR 
has  not  been  met.  On  the  other  hand,  an  N of 
1000  had  been  involved,  then  the  standard  error  of 
the  proportion  would  be  .009  ( . 27  ^ 1000,  Since 
2.79  times  .0086  is  .025,  there  is  99  per  cent 
confidence  that  the  value  is  between  . 895  and 
. 942. 

5.9  Determining  the  Required  Level  of  HR.  The  required  level 

of  HR  along  with  the  confidence  interval  and  level,  should 
be  specified  by  the  Government.  The  required  level  will 
depend  on  many  factors  including  but  not  limited  to;  the 
general  role  of  the  human  in  the  situation  under  considera- 
tion and  the  effects  of  human  failure  on  system/mission 
objectives,  the  nature  of  the  equipment/ subsystem/ system 
under  consideration,  and  the  complexity  of  the  tasks  the 
human  musr  perform. 

5.  10  Failure  of  the  Final  HR  Demonstration.  Satisfactory  per- 

formance on  the  final  HR  demonstration  is  never  certain 
prior  to  the  demonstration.  Prior  to  the  onset  of  the  HR 
demonstration,  contingency  plans  should  be  submitted  by 
the  contractor  for  implementation  in  the  event  of  failure 
to  meet  the  predetermined  HR  criterion.  Such  plans  should 
be  subject  to  approval  of  the  Government  and  probably 
involve  redesign  of  the  equipment/subsystem/system  rela- 
tive to  the  tasks  contributing  most  heavily  to  the  depressed 
HR  along  with  scheduling  of  an  additional  HR  demonstration 
at  a later  date.  Because  of  failure  of  the  HR  demonstration 
may  impact  equipment  acceptance,  the  earliest  possible 
scheduling  of  the  HR  demonstration  is  advised. 

5.11  Calculation  of  System  Reliability,  Availability,  and  VITTR. 

If  equipment  data  are  available  they  should  be  combined 
with  the  human  data  to  provide  overall  system  measures  of 
reliability,  availability,  and  WTTR. 
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System  Reliability  (SR).  System  reliability  should  be 
computed  using  the  formula:  (ER)  x (HR) 

where: 

ER  = equipment  reliability 
HR=  human  reliability 

System  Availability  (SA).  System  availability  should  be 
calculated  using  the  formula:  „ , / DT  USH 

" luT+DT  TAMH 

where:  ' 

DT=  Down  time 
lTT=  lTp  time 

USH=  Total  time  (in  man  hours)  demonstration  tasks 
are  extended  beyond  nominal  time 
TAMH=  Total  man  hours  available 


System  MTTR.  System  mean  time  to  repair  (SMTTR)  should 
be  calculated  using  the  formula: 


SMTTR  = 


TDT  + TFU 
NFU  + (NR) 


where: 

TDT=  total  equipment  down  time  during 
the  demonstration 

TFU=  the  total  amount  of  time  spent  in 
subtask  fix-up  attempts  during 
the  demonstration 

NFU=  the  total  number  of  subtask  fix-ups 
attempted  during  the  demonstration 

NR=  the  total  number  of  equipment  repairs 
during  the  demonstration 


For  example,  if  the  total  equipment  down  time  was  5 hours, 
the  time  spent  in  subtask  fix-up  was  2 hours,  a total  of  12 
fix-ups  was  performed, and  5 repairs  were  performed: 


5+2 


SMTTR  = 


12+5 


= .412 
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